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PREFACE 

One  of  the  major  missions  of  the  Rome  Air  Development 
Center  (RADC)  is  the  development  of  procedures  and  techniques  for 
improving  the  readiness  and  supportability  of  weapon  systems.  In 
support  of  this  mission,  RADC  has  sponsored  a  myriad  of  studies, 
analyses,  and  developments  that  have  resulted  in  techniques,  standards, 
and  procedures  aimed  at  reaching  this  goal. 

In  the  1980s  all  the  military  services  have  recognized  the 
importance  of  improving  the  diagnostic  capability  of  weapon  systems  as  a 
means  for  rapid  troubleshooting  and  repair  of  these  systems.  The 
research  and  development  efforts  conducted  by  RADC  are  reflected  in 
this  guide  by  synthesizing  the  results  of  these  many  efforts  and  filling 
gaps  to  provide  both  government  and  industry  with  a  compendium  of 
procedures  and  techniques  which  may  be  used  to  improve  the  fielded 
weapon  systems’  diagnostic  capability. 

Many  other  programs  have  made  the  contributions  that  are 
included  in  these  guides.  Information  has  been  freely  included  from 
various  military  service  and  industry  work.  Among  these  is  the  Air 
Force’s  Generic  Integrated  Maintenance  Diagnostics  Program  (GIMADS). 
The  Navy’s  Integrated  Diagnostic  Support  System  (IDSS)  and  the  Army’s 
Integrated  Diagnostics  in  the  Maintenance  Environment  (AIDME)  have 
also  made  valuable  contributions  to  this  guide.  In  this  manner,  material 
from  all  of  the  other  service  organizations  is  now  available  for  Joint 
Service  use. 

Three  (3)  guides  have  been  written  which  are  aimed  at  the 
following  users: 

o  Government  Program  Manger 

o  Contractor  Program  Manager 

o  System  Designer. 

Thus,  the  guidance  material  required  by  a  specific  user  will  be  included  in 
one  of  these  three  (3)  guides. 

It  is  believed  that  this  guidance  material  represents  a 
comprehensive  look  at  the  problems  in  fielding  a  satisfactory  diagnostic 
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capability  and  a  structured  system  engineering  approach  to  solving  these 
problems.  RADC  solicits  comments  on  this  guidance  material,  as  a 
means  for  improvements  in  the  coming  years. 

These  guides  have  been  prepared  under  contract  by  Giordano 
Associates,  Inc.,  with  subcontractor  assistance  from  Grumman 
Aerospace  Corporation  and  Rockwell  International. 
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INTRODUCTION 


WHY  ALL  THIS  WORRY  ABOUT  DIAGNOSTICS? 

Let’s  put  the  diagnostic  probiem  in  its  proper  perspective. 
You’ve  got  a  probiem  with  your  automobiie  and  you  turn  to  a  mechanic 
for  help.  Historically,  you  realize  that  the  problem  may  be  fixed  or  indeed, 
for  some  reason,  you  may  have  to  go  back  one  or  more  times  before  the 
problem  is  corrected,  or  you  give  up.  We  are  talking  about  automobiles. 
Automobiles,  which  a  manufacturer  has  produced  tens  of  thousands  of 
times,  have  a  historical  record  of  their  reliability  and  maintainability  and 
have  been  redesigned  and  reengineered  many  times. 

When  comparing  your  automobile  to  an  extremely  complex 
weapon  system  that  is  pushing  the  state  of  the  art  and  produced  in 
limited  numbers,  with  questionable  historical  data  on  their  operation,  one 
can  easily  understand  the  magnitude  of  the  problem. 

It  is  not  the  purpose  of  this  guide  to  provide  a  comprehensive 
discussion  of  the  diagnostic  problems,  but  rather  to  furnish  guidance  for 
circumventing  known  problem  areas.  However,  to  understand  the 
magnitude  of  the  problem  a  few  examples  follow. 

In  one  six-month  period,  at  one  F-16  tactical  fighter  wing,  over 
13,600  maintenance  manhours  were  reported  for  the  processing  of 
unnecessary  removals.  This  equals  about  20  people  just  working  on 
troubleshooting  these  "good"  items. 

A  DoD  Task  Force  on  Productivity  In  Support  Operations 
(1986)  found  that  20  to  50  percent  of  avionics  maintenance  actions 
resulted  in  removal  of  items  with  no  evidence  of  failures. 

The  deployment  of  an  avionics  Intermediate  shop  for  fighter 
aircraft  to  a  remote  location  can  require  anywhere  from  three  to  1 1  C- 
141B  equivalent  loads.  In  wartime,  there  just  will  not  be  enough  cargo 
aircraft  to  respond  to  this  need.  In  peacetime,  it’s  just  plain  costly. 

The  diagnostic  problem  is  not  unique  to  any  one  service,  nor  to 
any  one  type  of  weapon  system.  It  manifests  itself  throughout  the  military 
services.  The  problem  can  be  an  engineering  or  a  field  problem.  It  can 
be  a  man  or  a  machine  problem.  It  can  be  a  wartime  or  a  peacetime 
problem.  It  can  be  a  prime  system  or  a  supportability  problem.  The 
problem  manifests  Itself  In  different  ways  for  different  types  of  weapon 
systems,  but  the  consequences  are  all  the  same-long  times  to 


-v«- 


INTRODUCTION 


_ I 


troubleshoot,  removal  of  items  which  have  not  failed,  long  logistic  tails, 
and  an  overall  lack  of  confidence  in  the  entire  diagnostic  capability. 
Obviously,  the  result  is  lack  of  readiness  and  a  waste  of  dollars  and 
manpower. 

There  are  a  multitude  of  reports  which  adequately  describe  the 
problem.  Two  of  those  reports  give  a  comprehensive  picture  of  the 
problem  and  possible  solutions.  These  are: 

"Isolation  of  Faults  in  Air  Force  Weapon  and  Support  Systems," 
Committee  on  Isolation  of  Faults  in  Air  Force  Weapon  and  Support 
Systems,  Air  Force  Studies  Board,  Commission  on  Engineering  and 
Technical  Systems,  National  Research  Council. 

"Report  for  the  Department  of  Defense  on  the  Implementation 
of  Integrated  Diagnostics,"  prepared  by  the  National  Security  Industrial 
Association's  Integrated  Diagnostics  Working  Group,  September  1 984. 


HISTORICALLY  THE  HELDED  DIAQNOSTIC  CAPABIUTY  HAS  NOT 
UVED  UP  TO  THE  PROMISES 


Government  and  industry  must  share  the  responsibility  for  what 
has  happened  in  the  past.  On  the  government’s  side,  there  tends  to  be  a 
lack  of  knowledge  on  how  to  specify  what  is  needed  and  how  to  make 
Bare  the  government  gets  what  it  needs.  On  the  contractor’s  side  is  a 
lack  of  understanding  of  the  importance  of  fielding  a  satisfactory 
diagnostic  capability  and  still  maintain  schedule  and  cost  limitations. 
Hopefully,  the  series  of  guides  produced  under  this  program  will  help  to 
alleviate  this  situation. 

The  military  services,  as  well  as  the  Office  of  the  Secretary  of 
Defense,  understand  the  urgency  of  this  problem  and  have  established 
multimillion  dollar  programs  to  help  alleviate  this  situation,  both  from  a 
technology  and  a  management  perspective.  For  the  most  part,  these 
programs  are  generic-applicable  to  a  variety  of  weapon  systems. 


I  I 

DIAQNOSTIC  IMPROVEMENT  PROGRAMS  ARE  UNDERWAY 
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INTRODUCTION 


WHAT  ARE  THE  MAIN  THRUSTS  OF  THIS  GUIDE? 

Historically,  the  development,  tesi,  and  evaluation  of  the 
diagnostic  capability  for  a  weapon  system  often  have  been  more  or  less 
an  afterthought.  For  many  seemingly  logical  reasons,  satisfying  prime 
system  "performance"  requirements  has  taken  precedence  over 
diagnostic  requirements.  Simply,  the  goal  of  this  guide  is  to  promote  a 
realization  that  diagnostic  requirements  are  indeed  system  performance 
requirements,  and  thus  are  an  integ'al  part  of  weapon  systems’ 
development,  test,  and  evaluation.  Thus  this  guide  has  the  following 
attributes  in  relation  to  the  development  of  this  diagnostic  capability: 

o  Describes  a  system  engineering  approach 

o  Eliminates  stand-alone,  sometimes  conflicting,  diagnostic 
requirements. 

o  Requires  that  individual  diagnostic  elements  are  designed 
specifically  to  complement  each  other 


o  Places  responsibility  on  a  single  prime  contractor  to  meet 
diagnostic  requirements 

o  Provides  the  prime  contractor  with  design  latitude  and 
flexibility  to  meet  these  requirements 

o  Encourages  concurrent  design,  test,  and  evaluation  of  the 
entire  diagnostic  capability  with  the  prime  hardware  and 
software 

o  Allows  for  a  comprehensive  maturation  period  to  meet 
diagnostic  requirements. 


A  SYSTEMS  ENGINEERING  APPROACH  IS  KEYI 


DEFINITIONS 


WHAT  DO  WE  MEAN  BY  INTEGRATED  DIAGNOSTICS? 

Before  using  this  guide  it  is  imperative  that  you  understand  the 
definition  of  a  few  words.  The  first  term  is  "testability",  which  is  defined 
as  "a  design  characteristic  which  allows  the  status  of  the  unit  to  be 
confidently  determined  in  a  timely  manner."  It  includes  the  capability  to 
detect,  isolate  failures  and  minimize  false  alarms.  Therefore,  testability 
may  be  regarded  as  inherent  to  the  item’s  design. 

"Diagnostics"  is  defined  as  "the  hardware,  software  and/or  other 
documented  means  used  to  determine  a  malfunction  has  occurred  and  to 
isolate  the  cause  of  the  malfunction."  It  also  refers  to  "the  action  of 
detecting  and  isolating  failures." 


"Integrated  diagnostics"  is  defined  as  a  "structured  design  and 
management  process  to  achieve  the  maximum  effectiveness  of  a 
weapon  system’s  diagnostic  capability  by  considering  and  integrating  all 
related  pertinent  diagnostic  elements."  The  process  includes  interfaces 
between  design,  engineering,  testability,  reliability,  maintainability,  human 
engineering,  and  logistic  support  analysis.  The  goal  is  a  cost-effective 
capability  to  detect  and  unambiguously  isolate  all  faults  known  or 
expected  to  occur  in  weapon  systems  and  equipment  in  order  to  satisfy 
weapon  system  mission  requirements. 

"Diagnostic  capability"  refers  to  the  capability  of  the  system  to  detect 
and  isolate  faults,  utilizing  automatic  and  manual  testing,  maintenance 
aids,  technical  information  and  the  effects  of  personnel  and  training. 

"Diagnostic  element"  is  defined  as  one  part  of  the  diagnostic  capability 
(e.  g.,  ATE). 


"Diagnostic  Subsystem”  is  defined  as  all  the  diagnostic  elements  which 
constitute  a  weapon  system’s  diagnostic  capability. 

"Embedded  diagnostics"  is  defined  as  any  portion  of  the  weapon 
system’s  diagnostic  capability  which  is  an  integral  part  of  the  prime 
system  or  support  system.  "Integral"  implies  that  the  embedded  portion 
is  physically  enclosed  in  the  prime  system  and/or  permanently  attached- 
-physically  or  electrically. 
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"External  diagnostics"  is  defined  as  any  portion  of  the  weapon  system’s 
diagnostic  capability  which  is  not  embedded. 

It  is  important  to  understand  that  integrated  diagnostics  is  a  structured 
process  for  acquiring  a  diagnostic  capability. 


HOW  TO  USE  THIS  GUIDE 


For  a  better  understanding  of  the  various  diagnostic  activities 
that  take  piace  during  the  acquisition  and  deployment  of  a  weapon 
system,  a  Roadmap  has  been  prepared  for  your  use.  The  Roadmap 
depicts  all  of  the  diagnostic  activities  that  take  place  during  each  phase  of 
weapon  system  acquisition  and  deployment.  The  Roadmap  is  shown  in 
Figure  1 ,  with  inputs  and  outputs  for  each  activity.  This  Roadmap  gives 
the  reader  the  entire  picture  of  both  government  and  contractor 
diagnostic  activities  which  are  aimed  at  developing  an  adequate 
diagnostic  capability.  It  is  recognized  that  there  is  no  single  Roadmap  that 
can  apply  to  all  situations.  Thus  the  Roadmap  is  designed  with  multiple 
entry  points  to  provide  flexibility. 


THE  ROADMAP  GIVES  YOU  THE  BIG  PICTURE 


The  structure  of  the  guide  Is  built  around  this  Roadmap.  The 
activities  on  the  Roadmap  relate  to  seven  basic  requirements  listed  in 
Table  1.  This  document  is  structured  according  to  these  seven 
requirements. 

Reference  to  a  specific  requirement  is  shown  on  the  Roadmap, 
so  the  reader  can  quickly  relate  a  diagnostic  activity  on  the  Roadmap  to 
specific  guidance  information  contained  in  this  guide. 
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TABLE  1.  ESTABLISHED  REQUIREMENTS 


REQUIREMENT  # 

REQUIREMENT 

1 

ESTABLISHING  AND  JUSTIFYING  A  PROGRAM 

FOR  ACQUIRING  A  DIAGNOSTIC  CAPABILITY 

2 

ESTABLISHING  AND  ALLOCATING  DIAGNOSTIC 
REQUIREMENTS 

3 

DESIGNING  THE  DIAGNOSTIC  CAPABILITY 

4 

ANALYSIS  AND  ASSESSMENT  OF  THE  PERFORMANCE 

OF  THE  DIAGNOSTIC  CAPABILITY 

5 

CONDUCTING  DESIGN  REVIEWS 

6 

CONDUCTING  TEST  AND  EVALUATION 

7 

MATURATION  OF  THE  DIAGNOSTIC  CAPABILITY 

Each  one  of  these  basic  requirements  is  followed  by  detailed 
requirements  (e.  g.,  Requirement  1.2,  Preparing  an 
RFP/SOW/Specification).  Each  of  these  detailed  requirements  is  tied  to 
a  weapon  system  activity  and  a  weapon  system  acquisition  phase. 

The  three  guides  are  structured  in  essentially  the  same  way- 
•the  difference  being  that  the  guidance  material  supplied  is  tailored  for  the 
USER  of  each  specific  guide.  Each  requirement  is  highlighted  for  easy 
access  to  the  information  the  user  requires. 

Each  of  the  guides  contains  a  Lessons  Learned  Appendix 
(Appendix  A),  which  will  help  the  user  to  understand  how  this  guidance 
information  applies  to  real-world  acquisitions.  Appendix  B  is  a  list  of  the 
acronyms  used. 
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WEAPON  SYSTEM  CONCEPT  EXPLORATION 
ACQUISITION  PHASE 
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WEAPON  SYSTEM  RILL  SCALE  DEVELOPMENT  (CONTINUED) 
ACQUISITION  PHASE 
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HOW  TO  USE  THIS  GUIDE 


J 


THE  GUIDE  IS  STRUCTURED  TO  GIVE  THE  USER  EASY  ACCESS  TO 
THE  INFORMATION  REQUIRED 


It  is  recognized  that  a  guide  of  this  type  cannot  contain  all  the 
necessary  information  that  the  user  requires.  In  these  cases,  an  attempt 
has  been  made  to  cite  reference  documents,  such  as  military  standards, 
handbooks,  and  reports. 


AN  AUTOMATED  VERSION  OF  THIS  GUIDANCE,  PLUS  SOME 
COMPUTERIZED  TOOLS  ARE  AVAILABLE 


To  aid  in  the  use  of  these  guides,  a  computerized,  interactive 
version  of  all  three  guides  has  been  developed.  In  addition,  a  number  of 
computer-aided  tools,  suitable  for  inclusion  in  CAD  modules,  have  been 
included  for  use  by  design  engineers  in  the  implementation  of 
techniques,  procedures,  etc.  These  are  contained  in  the  Design 
Encyclopedia.  If  you  are  interested  in  obtaining  these  software  programs, 
you  may  contact  Rome  Air  Development  Center,  RADC/RBET,  Griffiss 
Air  Force  Base,  New  York,  13441-5700,  (telephone  number: 
3 15-330-4726,  AV  587-4726). 


PROGRAMMATIC 


REQUIREMENT  #1 


ESTABLISHING  AND  JUSTIFYING  A  PROGRAM  FOR  ACQUIRING  A  DIAGNOSTIC 
CAPABILITY 


OVERVIEW 


DoD  organizations  are  often  disappointed  at  the 
performance  of  a  weapon  system's  diagnostic 
capabiiity,  once  it  is  depioyed.  This 
disappointment  often  results  in  frustration  by  the 
user,  an  adversarial  relationship  between  the 
acquirer  and  the  producer,  and  costly  engineering 
changes.  The  fact  is  that  the  quality  of  the 
acquired  diagnostic  capability  is  a  two-way  street. 
In  the  case  of  the  government,  careful 
consideration  must  be  given  to  what  you  want  the 
contractor  to  deliver.  In  the  case  of  the  contractor, 
he  must  be  dedicated  to  producing  a  quality 
product.  Specifying  fault  detection  and  isolation 
requirements  is  a  difficult,  complex  job  for  the 
Government  Program  Manager.  Justifying  his 
program  to  a  higher  authority  in  clear,  concise 
terms  is  essential.  Establishing  realistic  and 
feasible  plans  for  satisfying  these  requirements  is 
a  prime  responsibility  of  the  Contractor  Program 
Manager,  implementing  these  plans  is  the 
responsibility  of  the  weapon  system  designer  and 
his  subcontractors.  Without  a  clear  understanding 
and  close  cooperation  among  these  people, 
production  of  a  less-than-satisfactory  diagnostic 
capabiiity  is  inevitable. 


TESTABILITY/ 

DIAGNOSTICS 

1 

PROGRAMMATIC  | 

REQUIREMENTS  | 

DESIGN  1 

ASSESSMENT  | 

REVIEWS  1 

EVALUATION  | 

MATURATION  | 

IMPORTANT  CONSIDERATIONS  TO  BE  ADDRESSED 
Reqmt. 

1.1  Assure  the  Statement  of  Need  (SON)  Is  written  in  mission  and 
performance  terms. 

1 .2  What  you  specify  In  an  RFP/SOW/SpecifIcation  is  what  you’ll  get 

1.3  Tie  diagnostic  capability  plans  to  the  system  engineering  plans. 

1 .4  Make  sure  that  SCPs  and  DCPs  address  diagnostic  capability  issues. 


1-1 


DIAGNOSTIC  CONSIDERATIONS  IN  PREPARING  AN  SON 


REQUIREMENT  #1.1 


WEAPON 
SYSTEM 
ACQ.  PHASE 


OPER.  CONCEPT  DEM/ 

REQMTS.  EXPLOR.  VAL 


FSD  prod/ 


WEAPON 

SYSTEM 

ACTIVITIES 


RFP 

PREP 


RFP 

PREP 


RFP 

PREP 


DIAGNOSTIC 

ACTIVITIES 


INPUTS  TO 
SON 


SON  INPUTS  REVALIDATED 


DIAGNOSTIC  ACTIVITY 

The  major  consideration  in  the  initiation  of  a  weapon  system  acquisition  program 
is  the  preparation  of  a  Statement  of  Need.  Each  of  the  services  has  its  own  designation 
for  this  document.  For  a  major  weapon  system,  DoD  Instruction  5000.2  designates  this 
document  as  a  "Mission-Need  Statement  (MNS)."  For  less-than-major  systems,  the 
services  use  other  terms,  such  as  "Operational  Requirements  Document"  and  "Required 
Operational  Capability."  It  is  important  that  these  documents  reflect  these  operational 
requirements  in  terms  which  can  be  properly  interpreted  to  produce  diagnostic 
requirements. 

PROCEDURE 


Each  of  the  military  services  has  issued  policy  directives  and  guidance  relating 
to  the  preparation  of  a  Statement  of  Need.  DoD  Instruction  5000.2  delineates  the  format 
for  an  MNS.  This  format  does  not  differ  appreciably  from  the  formats  used  for  less-than- 
major  new  starts,  thus  the  following  guidance  will  be  discussed  in  relation  to  the  MNS. 

The  SON  is  issued  prior  to  Concept  Exploration.  When  the  Concept  Exploration 
Phase  is  not  conducted,  the  SON  should  be  issued  prior  to  initiation  of  work.  In  addition, 
the  validity  of  the  SON  can  be  reevaluated  prior  to  the  initiation  of  Dem/Val,  FSD,  and 
Production. 


DIAGNOSTIC  CONSIDERATIONS  IN  PREPARING  AN  SON 


REQUIREMENT  #1.1 


GUIDANCE 


It  is  almost  as  important  to  ensure  what  should  not  be  put  into  an  SON  as  what 
should  be  put  into  an  SON.  Diagnostic  requirements  must  reflect  a  threat  and  mission  and 
operational  needs,  plus  certain  constraints  put  on  the  weapon  system,  such  as  resource 
limitations.  At  the  initiation  of  a  weapon  system  development,  it  is  important  that  the 
Government  Program  Manager  not  be  limited  by  establishing  premature  diagnostic 
requirements,  such  as  a  certain  percent  fault  detection/fault  isolation  to  a  given  unit,  or  an 
MTTR.  Rather,  the  contractor  should  be  given  the  flexibility  to  derive  the  diagnostic 
requirements  from  mission  needs,  such  as  sortie  rates,  mobiiity  requirements,  and  the 
mission  scenario. 

The  format  for  the  MNS  is  as  follows: 

FORMAT  FOR  MISSION-NEED  STATEMENT 

A.  Defense  Guidance  Bement 

Identify  the  element  of  Defense  Guidance  to  which  the  acquisition  responds. 

B.  Mission  and  Threat 

Identify  the  OUSO(A)  mission  area  (numbers  and  title)  and  describe  the  mission  area  need. 
Also  Identify  the  related  Defense  Guidance  mission  area  (numbers  and  title).  Discuss  the 
D I  A- validated  projected  threat  and  the  shortfalls  of  existing  systems  or  facilities  in  meeting 
the  threat.  Comment  on  the  timing  of  the  need  and  the  general  priority  of  this  acquisition 
relative  to  others  in  this  mission  area. 

C.  Alternative  Concepts 

Describe  known  alternatives  that  will  be  considered  during  the  Concept 
Exploration/Definition  Phase  (including  use  of  an  existing  U.  S.  or  Allied  military  or 
commercial  system  or  product  improvements  of  an  existing  system). 


D.  Cooperative  Opportunities  Document 

Indicate  whether  or  not  a  program  addressing  a  similar  need  is  in  development,  or 
production,  by  one  of  the  Allied  nations.  If  so,  indicate  whether  that  program  could  satisfy 
the  military  requirements  of  the  United  States.  Assess  the  advantages  and  disadvantages 
of  seeking  to  structure  a  cooperative  development  program  with  one  or  more  of  the  Allied 
nations.  Include  a  recommendation  as  to  whether  the  Department  oi  Defense  should 
explore  the  feasibility  and  desirability  of  a  cooperative  development  program  with  one  or 
more  of  the  Allied  nations. 

E.  Technology  Involved 

For  known  alternatives,  discuss  maturity  of  the  technology  planned  for  the  selected  system 
design  and  manufacturing  processes,  with  particular  emphasis  on  remaining  areas  of  risk. 
Discuss  projected,  sustained  industrial  support  for  the  planned  baseline  technology  and  the 
projected  need  for  off-shore  production  and  technology  support.  Discuss  system  and 
subsystem  prototyping  considerations. 
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F.  Funding  Implications 

Discuss  affordability,  Including  the  level  of  funding  the  component  is  willing  to  commit  to 
satisfy  the  need,  and  whether  this  level  will  fully  fund  the  program  over  the  approved 
Defense  Five-Year  Defense  Program  (FYDP). 

Q.  Constraints 

Describe  applicable,  key  boundary  conditions  for  satisfying  the  need,  such  as  reliability, 
maintainability,  and  survivsdiility;  logistics  support,  mapping,  charting,  and  geodesy  support, 
manpower,  personnel,  training,  and  safety  constraints;  computer  resources;  standahtizatlon 
or  interoperability  within  NATO  or  other  DoD  components;  and,  critical  materials  and 
Industrial  base  considerations.  Provide  justification  for  any  proposed  nonuse  of  the  metric 
measurement  system  (DoD  Directive  3120.18  (reference  (r)). 

H.  Acquisition  Strategy 

Provide  a  summary  of  saliertt  elements  of  proposed  acquisition  strategy,  such  as  program 
structure,  competition,  contracting  approach,  and  acquisition  streamlining. 

Weapon  system  mission  and  threat  needs,  along  with  constraints,  are  the  sole 
basis  for  establishing  diagnostic  requirements  during  the  Concept  Exploration  Phase. 
Therefore,  it  is  important  that  these  needs  are  written  in  a  manner  which  can  subsequently 
establish  these  diagnostic  requirements.  Examples  of  needs  In  the  SON  which  initially 
drive  diagnostics,  are: 


o  Mobility 
o  Basing 
o  Mission  scenario 
o  Length  of  mission 
o  Cost  constraints 
0  Manpower  limitations. 
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REQUIREMENT  #1.1 


CHECKLIST 

^  Does  the  SON  adequately  address  mission  and  threat 
and  refrain  from  prematurely  including  diagnostic 
requirements? 

of  Are  the  constraints  that  effect  the  establishment 
of  diagnostic  requirements  (e.g.,  mobility,  basing, 
cost,  manpower)  cited? 


PREPARING  AN  RFP,  SOW,  SPECIFICATION 


REQUIREMENT  #1.2 


WEAPON 
SYSTEM 
ACQ.  PHASE 


WEAPON 

SYSTEM 

ACTIVITY 


OPER. 

REQMTS. 


CONCEPT 

EXPLOR. 


PROD/ 

DEPL 


DIAGNOSTIC 

ACTIVITIES 


DIAGNOSTIC  INPUTS  TO  RFP/SOW/SPEC 


DIAGNOSTIC  ACTIVITY 

Clear,  concise,  and  feasible  provisions  must  be  inserted  into  the  Request  for 
Proposal  (RFP),  the  Statement  of  Work  (SOW),  and  the  System  Specification,  as  a  means 
for  assuring  that  the  contractor  and  his  subcontractors  have  a  clear  understanding  of  what 
is  required  of  the  diagnostic  capability. 

PROCEDURE 


One  of  the  initial  tasks  which  must  be  undertaken  by  the  Government  Program 
Manager,  at  the  beginning  of  each  acquisition  phase,  is  the  development  of  the  Request 
for  Proposal,  which  will  subsequently  lead  to  a  contractual  document.  For  the  Concept 
Exploration  Phase,  normally  the  RFP  contains  a  Statement  of  Work  without  an  associated 
weapon  system  specification.  The  specification  is  normally  invoked  no  sooner  than  the 
Demonstration  and  Validation  Phase.  It  is  normally  written  by  the  contractor  as  a  data 
deliverable,  with  final  review  by  the  Government  Program  Manager.  The  requirements  for 
this  diagnostic  capability  must  appear  in  a  variety  of  places  throughout  these  documents 
to  assure  the  acquisition  of  a  satisfactory  diagnostic  capability.  For  the  Concept 
Exploration  Phase,  these  requirements  are  general  in  nature  and  allow  the  maximum 
flexibility  for  the  contractor  to  do  his  job.  As  the  weapon  system  design  proceeds,  these 
requirements  become  more  and  more  specific.  The  thrust  and  content  of  the  provisions 
contained  In  these  documents  varies,  depending  on  the  acquisition  strategy  developed  by 
the  Government  Program  Manager,  the  phase  in  which  these  documents  are  invoked,  and 
the  size  and  complexity  of  the  weapon  system. 


PREPARING  AN  RFP,  SOW,  SPECIFICATION  REQUIREMENT  #1.2 

GUIDANCE 


No  guidance  on  the  content  and  form  of  what  should  be  included  in  an  RFP  and 
System  Specification  can  be  made,  which  is  applicable  to  every  system  or  equipment. 
Thus  the  information  which  follows  must  be  perceived  as  examples  of  how  to  best  specify 
a  weapon  system’s  diagnostic  capability.  Tailoring  is  a  must. 

Preparing  an  RFP: 

Diagnostics  impacts  a  number  of  sections  within  an  RFP,  as  shown  in  the 
following  paragraphs. 

Special  Contract  Requirements  (Section  H)  -  Contractor  incentives  and  warranties  are 
contained  in  this  section  of  the  RFP.  The  type  and  content  of  these  incentives  and 
warranties  are  almost  limitless,  depending  on  the  innovation  of  the  RFP  writer.  The 
Defense  System  Management  College  has  published  a  warranty  handbook,  which  is  a 
reference  guide  for  use  by  DoD  managers  in  developing,  applying  and  administering 
warranties.  This  guide  contains: 

o  Warranty  law  and  DoD  Policy 
o  Warranty  concepts  and  issues 
o  Warranty  selection  and  structure 
o  Warranty  development 
o  Warranty  administration 
o  Warranty  cost  benefit  analysis 
o  Case  examples  of  warranties. 

Several  of  these  warranties  are  applicable  to  the  fault  detection,  fault  isolation 
process.  These  deal  with  reliability  improvements,  MTBF  guaranties,  availability 
guaranties,  and  logistic  support  costs  guaranties.  Copies  of  this  document  can  be 
obtained  from  the  Defense  System  Management  College,  who  is  the  controlling  agency  for 
this  handbook. 


Instructions  to  Offerors  (Section  L)  -  Emphasis  must  be  placed  on  introducing  the 
concept  of  integrated  diagnostics.  Although  no  standard  format  exists  for  this  section  of 
the  RFP,  this  section  must  address  the  need  for  managerial  and  technical  Information 
relative  to  integrated  diagnostics  and  the  meeting  of  the  diagnostic  requirements. 
Explanation  is  required  so  that  the  contractor  understands  that  integrated  diagnostics 
interfaces  with  logistics,  reliability,  maintainability,  testability,  human  engineering,  and 
safety  requirements.  The  contractor  must  be  alerted  that  he  will  be  judged  on  how  well 
this  integration  is  planned  and  how  advanced  technology  will  facilitate  this  integration. 

Automation  of  the  diagnostic  design  process  should  be  addressed,  because  it 
can  provide  for  a  more  efficient  and  effective  design  process.  It  is  not  the  government’s 
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job  to  dictate  the  use  of  these  design  tools,  but  rather  to  encourage  their  use.  This  can  be 
accomplished  by  adding  provisions  to  the  Instructions  to  Offerors  relating  to: 

o  A  discussion  of  design  aids  which  will  facilitate  the  design  and  integration  of 
the  diagnostic  capability  into  the  system  engineering  process 

o  The  development  and  use  of  a  diagnostic  data  base  which  supports  the 
application  of  these  tools 

o  Identification  of  how  automation  will  reduce  risk  in  the  design  of  the 
diagnostic  capability 

o  Means  for  providing  the  government  with  appropriate  documentation  for 
understanding  and  validating  the  output  of  the  automation  process 

o  Additional  information  on  motivating  the  contractor  to  utilize  automation  in 
the  diagnostic  design  process  is  included  in  the  Computer-Aided  Acquisition 
and  Logistic  Support  (CALS)  Implementation  Guide.  This  guide  will  be 
issued  as  a  military  handbook.  Copies  can  be  obtained  from  the  CALS 
Policy  Office,  Office  of  the  Secretary  of  Defense.  Included  in  this  guide  are 
means  for  tailoring  CDRLs  and  DIDs  to  encourage  the  use  of  design 
automation. 

Evaluation  Factors  for  Award  (Section  M)  -  This  section  must  be  written  to  assure  that 
the  proposal  writer  understands  that  integrated  diagnostics  and  diagnostic  requirements 
will  have  a  significant  impact  on  the  selection  of  a  contractor.  The  evaluation  factors 
should  reflect  the  diagnostic  content  of  the  Instructions  to  Offerors  (Section  L)  from  both 
technical  and  management  points  of  view.  Thus,  the  recognition  that  testability  and 
integrated  diagnostics  is  part  of  the  system  engineering  process  must  be  described  in  this 
section,  along  with  the  ability  to  utilize  advanced  technology  in  solving  this  problem.  From 
a  management  viewpoint,  the  evaluation  should  emphasize  the  need  for  a  single  person 
being  responsible  for  the  entire  diagnostic  capability,  often  requiring  a  person  full-time. 

In  addition  to  having  the  evaluation  factors  reflect  the  content  of  the  Instructions 
to  Offerors,  several  other  evaluation  factors  are  of  importance.  These  include: 

o  The  amount  and  type  of  specialized  education  and  training  given  to  both 
contractor  program  managers  and  designers  which  relate  to  testability  and 
integrated  diagnostics  process 

o  The  independent  research  and  development  conducted  by  the  contractor 
which  relates  to  testability  and  diagnostic  design  tool  development  and 
demonstrations  of  the  integration  of  diagnostic  elements 
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o  Methods  and  scheduling  to  assure  the  concurrent  delivery  and  evaluation  of 
the  entire  diagnostic  capability  with  the  prime  system  itself 

0  How  the  diagnostics  for  both  GFE  and  CFE  will  be  addressed  by  the 
contractor  to  assure  that  overall  system  diagnostic  requirements  are  met 

0  The  quality  of  the  diagnostic  maturation  program  proposed  by  the  contractor. 

Preparing  Statements  of  Work 

The  Statement  of  Work  is  key  to  acquiring  a  satisfactory  diagnostic  capability. 
Experience  has  shown  that  the  contractor  must  be  given  freedom  to  meet  mission  needs 
within  operational  constraints  in  a  way  which  promotes  innovative  design.  Design 
tradeoffs  are  required  to  properly  allocate  FD/FI  requirements.  Early  design  verification  is 
essential,  so  that  modifications  to  the  diagnostic  capability  design  can  be  made  in  a  timely 
manner.  The  result  will  be  a  better  capability  at  a  lower  cost,  while  avoiding  costly  retrofits, 
design  errors,  and  late  delivery. 

The  Statement  of  Work  varies,  depending  on  which  weapon  system  acquisition 
phase  is  being  addressed.  Four  sample  Statements  of  Work  follow:  one  each  for  Concept 
Exploration,  Demonstration  and  Validation,  Full-Scale  Development,  and  Production. 
These  samples  are  introduced  as  a  baseline.  Tailoring  is  encouraged.  In  summary,  the 
principal  attributes  of  these  Statements  of  Work  include: 

1 .  An  engineering  analysis  (including  gathering  of  field  data)  from  a  previously 
fielded  weapon  system(s)  to  determine  diagnostic  capability  performance 
deficiencies  experienced 

2.  Identification  of  specific  risk  areas  which  require  design  attention 

3.  A  requirement  for  preparation  and  implementation  of  a  diagnostic  capability 
maturation  plan,  including  assets  required,  activities  required,  and  data 
collection 

4.  Thorough  analysis  of  the  design  of  the  embedded  diagnostics  to  be 
completed  by  CDR 

5.  Design  analysis  and  specification  of  the  external  diagnostic  capability, 
including  overlap,  by  CDR 

6.  A  requirement  for  demonstration  of  the  diagnostic  capability,  including  a 
thorough,  statistically  valid  sample  in  selected  areas  of  the  system. 

The  following  sample  SOWs  address  diagnostic  tasks  which  can  be  included  in 
a  specific  portion  of  the  SOW  Requirements  Section.  When  writing  the  SOW,  it  should  be 
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recognized  that  diagnostics  plays  a  part  in  many  other  subsections  of  the  SOW  (e.  g., 
battle  damage  assessment,  faulHolerant  system  architecture,  safety,  in-flight  monitoring, 
configuration  management,  training,  technical  data,  test  equipment).  Thus,  when  writing 
these  subsections,  the  interface  with  diagnostics  should  be  kept  in  mind. 
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I  .  I  r 

CONCEPT  EXPLORATION  PHASE 
(Sample  SOW) 

DIAGNOSTIC  APPROACH 

The  contractor  shall  define  the  testability/diagnostic  approach  provided  for  the 
maintenance  of  each  system  alternative.  In  accomplishing  this,  the  contractor  shall: 

1.  Establish  overall  diagnostic  design  objectives,  strategies,  goals,  thresholds,  and 
constraints  which  support  mission  requirements  and  operational  constraints  in  support  of 
the  logistic  support  analysis  process  of  MIL-STD-1 388-1  and  the  system  engineering 
process  of  MIL-STD-499.  These  include: 

a.  Translation  of  weapon  system  mission  and  performance  requirements  into 
diagnostic  requirements  for  each  level  of  maintenance  which  support  the 
mission  scenario 

b.  Establishment  of  requirements  which  allow  for  diagnostic  growth  as  design 
proceeds  through  the  weapon  system  acquisition  phases 

c.  Identification  of  diagnostics-related  constraints  driven  by  operational 
constraints  of  the  system 

d.  Identification  of  technology  advancements  which  can  be  exploited  in  system 
development  and  diagnostic  elenrant  development  and  which  have  the 
potential  for  increasing  diagnostic  effectiveness;  reducing  the  requirement  for 
maintenance;  reducing  test  equipment,  technical  manuals  and  manpower, 
and  skill-level  requirements;  reducing  diagnostic  costs;  or  enhancing  system 
availability 

e.  Identification  of  existing  and  planned  diagnostic  resources  (e.  g.,  family  of 
testers,  maintenance  aids),  which  have  potential  benefits.  Identification  of 
resource  limitations 

f.  Identification  of  diagnostics  problems  on  similar  systems  which  should  be 
avoided. 

2.  Define  what  constitutes  a  system  failure  and  establish  deferred  maintenance, 
performance  and  safety  monitoring,  embedded  diagnostic  and  external  diagnostic 
objectives  for  the  new  system  at  the  system  and  subsystem  levels.  Identify  the  risks  and 
uncertainties  involved  in  achieving  the  objectives  established. 

3.  Establish  BIT,  test  equipment,  technical  information,  and  maintenance  manpower  and 
skill-level  constraints  for  the  new  system  for  inclusion  in  System  Specifications  or  other 
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requirements  documents.  These  constraints  shali  include  both  quantitative  and  qualitative 
factors. 

4.  Evaluate  alternative  diagnostic  concepts  to  include  varying  degrees  of  BIT,  manual  and 
automatic  testing,  technical  information  format  and  delivery  systems,  personnel  and 
training,  along  with  deferred,  preventive,  and  scheduled  maintenance  concepts,  and 
identify  the  selected  concept.  The  evaluation  shall  include: 

a.  A  determination  of  the  sensitivity  of  system  mission  performance  and 
readiness  parameters  to  variations  in  key  diagnostic  element  parameters 

b.  A  determination  of  the  sensitivity  of  life  cycle  costs  to  variations  in  diagnostic 
element  parameters 

c.  An  estimation  of  the  manpower  and  personnel  implications  of  alternative 
diagnostic  concepts  in  terms  of  direct  maintenance  manhours  per  operating 
hour,  job  classification,  skill  levels,  and  experience  required  at  each  level  of 
maintenance 

d.  An  estimation  of  the  risk  associated  with  each  concept. 

DIAGNOSTIC  CAPABILITY  PLANNING 

The  contractor  shall  develop  a  Diagnostic  Capability  Program  Plan  which 
describes  how  the  program  will  be  conducted.  The  program  plan  shall  be  included  as  part 
of  the  System  Engineering  Management  Plan  (SEMP).  The  plan  describes  the  time 
phasing  of  each  task  included  in  the  contractual  requirements  and  its  relationship  to  other 
tasks. 


The  plan  shall  address,  but  not  be  limited  to,  the  following  requirements: 

a.  Identify  a  single  organizational  element  within  the  performing  activity  which 
has  overall  responsibility  and  authority  for  implementation  of  the  program. 
Establish  analyses  and  data  interfaces  among  the  organizational  eler^ents 
responsible  for  each  of  the  elements  of  the  diagnostic  capability  and  other 
related  elements. 

b.  Develop  a  process  by  which  diagnostic  requirements  are  integrated  with 
other  design  requirements  and  disseminated  to  design  personnel  and 
subcontractors.  Establish  controls  for  assuring  that  each  subcontractor’s 
diagnostic  practices  are  consistent  with  overall  system  or  equipment 
requirements. 

c.  Identify  diagnostic  design  guides,  analysis  models  and  procedures  to  be 
imposed  upon  the  design  process.  Plan  for  the  review,  verification,  and 
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utilization  of  diagnostic  data  submissions.  Explain  how  computer-aided 
design  tools  will  be  utilized  in  this  process. 

DIAGNOSTIC  PROGRAM  REVIEW 

As  part  of  the  System  Requirements  Review  (SRR),  conduct  a  review  of  the 
above  required  diagnostics  approach  in  relation  to  the  above  requirements.  Conduct  and 
document  diagnostic  design  reviews  with  performing  activity  personnel  and  with 
subcontractors.  Coordinate  and  conduct  diagnostic  reviews  in  conjunction  with  reliability, 
maintainability,  human  engineering,  and  logistic  support  reviews,  whenever  possible. 
Utilize  MIL-STD-1521  and  program  review  criteria  contained  in  MIL-STDs  470,  785,  2165, 
and  1388-1  as  guidance. 


RECOMMENDED  LIST  OF  DATA  DELIVERABLES 
AND  CANDIDATE  DIDs 

o  Current  Diagnostic  Capability  Baseline  Analysis  Results  {DI-S-71 1 6) 
o  Recommended  System  and  Subsystem  Level  Diagnostic  Performance  and 
Approaches  (Specification  preparation  is  optional)  {DI-CMAN-80008) 
o  Diagnostic  Implementation  Feasibility/Risk  Reduction  Proposals  (DI-T-7199) 
o  Documented  results  of  diagnostic  assessment  as  an  integral  part  of  System 
Requirements  Review  documentation  (see  MIL-STD-1521)  (DI-A-7089) 

These  candidate  DID’S  should  be  tailored  by  CDRLs. 
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DEMONSTRATION  AND  VALIDATION  PHASE 
(Sample  SOW) 

Diagnostic  Specification  Development 

The  contractor  shall  perform  detailed  comparability  and  design  analysis  and  risk 
reduction  efforts  necessary  to  develop  a  specification  provision  which  allocates 
testability/diagnostic  requirements.  These  will  be  used  for  fault  detection/isolation,  repair 
verification,  performance  or  safety  monitoring,  and  damage  assessment.  This  will  enable 
the  weapon  system  to  meet  maintenance  and  operational  goals  with  a  minimum  of 
unnecessary  removals.  Diagnostic  capabilities  shall  be  selected  from  design  techniques 
(including  built-in  test,  fault  tolerance,  status  monitoring,  partitioning,  test  points):  external 
hardware  (e.  g.,  automatic  and  manual  test  equipment  and  maintenance  aids);  technical 
information  (e.  g.,  technical  manuals,  information  systems,  and  operator  displays);  and 
training  (e.  g.,  formal  schooling,  on-the-job  training).  The  capabilities  selected  may  be 
designed  into  the  system  as  part  of  the  system  and/or  provided  separately  to  maintenance 
personnel,  as  required,  to  meet  mission  and  level-of-repair  objectives. 

Based  on  the  results  of  the  analyses  and  risk  reduction  efforts,  the  contractor 
shall  specify  the  diagnostic  capabilities  tr'  be  provided  with  the  system  at  each  level  of 
maintenance  and  how  these  will  be  allocated.  This  includes; 

a.  Mode  of  operation  (e.  g.,  status  monitoring)  in  areas  where  there  is 
diagnostic  ambiguity  or  overlap 

b.  Operational  test  strategies,  fault  tolerance,  prognostics,  and  fault  model 
assumptions 

c.  Performance  in  terms  of  mean  time  to  diagnose,  fault  coverage,  false 
alarms,  and  false  removals 

d.  Physical  and  functional  equipment  partitioning  requirements 

e.  Physical  (weight,  volume)  and  functional  (%  memory)  limitations 

f.  Diagnostic  capability  interface  requirements 

g.  Options  for  augmenting  government-furnished  equipment  (GFE)  diagnostic 
capabilities 

h.  Reliability  of  the  BIT  and  external  diagnostic  hardware. 
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Detailed  Diagnostic  Comparison  Analysis 

As  part  of  the  development  of  the  diagnostic  specification  provisions,  the 
contractor  shall  perform  a  comparison  analysis,  using  the  baseline  fielded  system  at  each 
level  of  field  maintenance,  to  include  analysis  of  the  causes  of  diagnostic  times, 
undetected  faults,  "false  alarms,"  and  "false  removals"  which  are  judged  to  be  excessive. 
The  contractor  shall  identify,  to  the  greatest  extent  practicable,  the  sources  of  these 
causes  and  describe  how  the  new,  proposed  system  design  and  diagnostic  capabilities 
will  result  in  improvements.  As  a  minimum,  this  analysis  should  characterize  whether  the 
causes  of  diagnostic  problems  are  inherent  to  the  design  (e.g.,  reliability,  maintainability, 
partitioning,  connectors),  due  to  maintenance  procedures,  lack  of  "vertical"  testability  (e. 
g.,  cone  of  tolerance,  compatibility  between  levels  of  maintenance),  or  transients. 

The  contractor  will  provide  quantitative  assessments  of  diagnostic  capabilities 
identifying  current  capabilities,  extrapolations  to  proposed  capabilities,  and  the  engineering 
analysis  that  is  the  basis  for  the  extrapolation.  The  contractor  shall  determine  where  there 
are  overlaps  or  ambiguities  in  diagnostic  capabilities  used  for  maintenance  of  fielded 
systems  and  how  these  will  be  addressed  for  the  proposed  system.  When  deficiencies  in 
the  GFE  preclude  meeting  the  diagnostic  requirements,  the  contractor  shall  develop 
alternatives.  In  addition,  the  contractor  shall  identify  the  weight  and  volume  of  the  major 
external  test  equipment,  type  and  extent  of  technical  information,  maintenance  skill  levels 
and  training  requirements  for  currently  fielded  systems,  and  provide  an  estimate  of  these 
quantities  for  the  proposed  diagnostic  capability  and  explanation  of  the  basis  for  this 
estimate. 

Diagnostic  Risk  Reduction 

As  part  of  the  design,  prototype,  test,  and  demonstration  activities  proposed  (the 
basis  of  the  proposal  shall  be  risk  areas  identified  in  Concept  Exploration),  the  contractor 
shall  determine  the  feasibility  of  achieving  diagnostic  capability  performance 
improvements. 

Diagnostic  Maturation  Plan 

This  plan  shall  include  the  contractor’s  proposal  for  next  phase  activities  in  the 
areas  of  analysis,  growth,  and  demonstration  of  the  entire  diagnostic  capability  (hardware 
and  software).  The  plan  shall  also  include  the  resources  required  for  maturation  activities 
(e.  g.,  prime  hardware,  laboratory  facilities). 

The  contractor  shall  design  a  diagnostic  data  system  which  extends  from 
Demonstration  and  Validation  through  Full-Scale  Development,  Production,  and 
Deployment.  The  data  system  shall  be  designed  so  that  the  performance  of  the  diagnostic 
capability  can  be  ascertained  at  any  point  during  the  acquisition,  production,  and 
deployment  of  the  weapon  system.  It  shall  be  compatible  with  the  established  DoD  data 
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system,  which  will  be  employed  after  the  maintenance  of  the  weapon  system  becomes  the 
responsibility  of  the  government. 

Testability,  Preliminary  Design 

The  contractor  shall  apply  testability  design  criteria  to  the  design  of  selected 
high-risk  items,  in  accordance  with  MIL-STD-21 65,  Task  202.2.1.  The  testability  design 
criteria  to  be  considered  shall  include  selective  implementation  of  system-level  diagnostic 
strategies,  partitioning  to  enhance  fault  isolation,  initialization  of  circuitry  under  test  control, 
module  interface  for  test  access  and  control,  circuit  controllability  and  observability,  parts 
selection,  test  point  placement,  and  BIT  fault  detection  approaches. 

Diagnostic  Capabiiity  Pianning 

The  contractor  shall  develop  a  Diagnostic  Capability  Program  Plan  which 
describes  how  the  program  will  be  conducted.  The  program  plan  shall  be  included  as  part 
of  the  SEMP.  The  plan  describes  the  time  phasing  of  each  task  included  in  the 
contractual  requirements  and  its  relationship  to  other  tasks.  Diagnostic  issues  which  relate 
to  reliability,  maintainability,  logistics,  human  engineering,  safety,  etc.,  should  be 
addressed  in  those  individual  plans.  The  plan  shall  address,  but  not  be  limited  to,  the 
following  requirements: 

1 .  identify  a  single  organizational  element  within  the  performing  activity  which 
has  overall  responsibility  and  authority  for  implementation  of  the  program. 
Establish  analyses  and  data  interfaces  among  the  organizational  elements 
responsible  for  each  of  the  elements  of  the  diagnostic  capability  and  other 
related  elements. 

2.  Develop  a  process  by  which  diagnostic  requirements  are  integrated  with 
other  design  requirements  and  disseminated  to  design  personnel  and 
subcontractors.  Establish  controls  for  assuring  that  each  subcontractor's 
diagnostic  practices  are  consistent  with  overall  system  or  equipment 
requirements. 

3.  Identify  diagnostic  design  guides,  analysis  models  and  procedures  to  be 
imposed  on  the  design  process.  Plan  for  the  review,  verification,  and 
utilization  of  diagnostic  data  submissions.  Explain  how  computer-aided 
design  tools  will  be  utilized  in  this  process. 

Diagnostic  Program  Reviews 

As  part  of  the  System  Design  Review  (SDR)  conduct  a  review  of  the  diagnostic 
specification  provisions,  the  diagnostic  capability  program  planning,  and  the  preliminary 
testability  design.  Coordinate  and  conduct  diagnostic  reviews  in  conjunction  with 
reliability,  maintainability,  human  engineering,  and  logistic  support  reviews,  whenever 
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possible.  Utilize  MIL-STD-1521  and  program  review  criteria  contained  in  MlL-STDs  470, 
785,  2165,  and  1388-1  as  guidance. 

RECOMMENDED  UST  OF  DATA  DELIVERABLES 
AND  CANDIDATE  DIDs 

o  Proposed  Diagnostic  System  and  Subsystem  Performance  Specification 
Provisions  and  Allocations  and  Design  Requirements  (DI-CMAN  80008  & 
80025) 

o  Diagnostic  Maturation  Plan  to  include  System  Testing,  Design  Analysis,  and 
Data  Collection  (DI-R-21597) 
o  Results  of  Risk  Reduction  Tasks(DI-T-71 99) 
o  Results  of  Comparison  Analysis  (DI-S-71 1 6) 
o  Testability  Analysis  Report,  including  the  following: 

-  Description  of  system  built-in  test  functional  design  and  system  partitioning 
used  to  enhance  testing 

-  For  each  item  to  be  included  in  this  analysis,  a  description  of  testability 
features  incorporated  (compatibility,  observability,  controllability,  partitioning, 
etc.),  BIT  functional  design,  and  BIT  interfaces  to  system  BIT  and  external 
test.  (DI-T-7199) 

o  Documented  results  of  diagnostic  assessment  as  an  integral  part  of  System 
Design  Review  documentation  (see  MIL-STD-1521).  (DI-A-7089) 

These  candidate  DIDs  should  be  tailored  by  CDRLs. 
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FULL-SCALE  DEVELOPMENT  PHASE 
(SaiT^ie  SOW) 

Design  of  the  Diagnostic  Capability 

As  part  of  the  system  design,  the  contractor  shall  incorporate  embedded 
diagnostics  and  testability  features  in  the  system  and  provide  external  diagnostics 
capabilities  which  will  satisfy  the  diagnostic  capability  performance  requirements  contained 
in  the  system  specification. 

Diagnostic  Design  Analysis 

The  contractor  shall  implement  a  structured  design  analysis  process  to  assess 
in  detail  the  ability  of  the  diagnostic  capability  design  to  meet  the  system  diagnostic 
performance  specification  (e.  g.,  fault  coverage,  mean  time  to  diagnose,  false  removal, 
etc.);  analyze  the  inherent  testability  of  the  preliminary  design;  identify  the  areas  where  the 
primary  means  of  diagnostics  may  lead  to  an  ambiguous  result  and  ways  the  ambiguity 
will  be  resolved;  identify  areas  where  there  is  a  redundant  (overlapping)  diagnostic 
capability  planned  to  be  provided;  and  verify  that  the  detailed  design  of  diagnostics  is  in 
accordance  with  the  functional  allocation  established  during  the  previous  program  phase. 
The  analytical  task  shall  be  performed  and  delivered  as  described  below: 

o  Design  Analysis  of  Diagnostics  Built  In  the  System 

The  contractor  shall  complete  a  structured  analysis  of  system  design 
implementation  to  identify  the  functional  areas  in  which  diagnostics 
capabilities  allocated  in  the  previous  phase  to  be  built  into  the  system 
provide  an  unambiguous  capability  to  detect  or  isolate  a  fault  to  the 
appropriate  replaceable  unit  at  each  level  of  maintenance.  As  a  minimum, 
the  design  analysis  shall  be  based  on  maintenance  dependency  models  (or 
their  equivalents)  at  the  system  level,  to  quantify  the  degree  of  ambiguity  at 
the  lowest  replaceable  assembly  and  an  assessment  of  the  inherent 
testability  of  the  design  (Tasks  201,  202,  203  of  MIL-STD-2165),  where  there 
are  large  areas  of  ambiguity.  In  electronic  assemblies  consisting  of  digital 
logic,  the  dependency  model  analysis  shall  be  augmented  with  100%  fault 
simulation  for  selected  samples.  In  addition,  the  contractor  shall  prepare  a 
worst  case  analysis  of  design  or  tolerance  margins  at  each  BIT  sensor  and 
test  point  for  the  system.  As  a  result  of  these  analyses,  an  assessment  shall 
be  provided  of  the  capability  of  the  built-in  diagnostics  to  meet  the  fault 
coverage  specification,  identification  of  specific  system  areas  where  there  is 
ambiguous  fault  detection  or  isolation,  and  assessment  of  the  capability  to 
limit  the  false  alarms  and  false  removals.  The  assessment  shall  identify  the 
projected  causes  of  false  alarms  and  false  removals  and  ambiguities  in 
terms  attributable  to  equipment  design  mechanization  (including  partitioning, 
test  point  placement,  BIT  limitations),  transients,  or  maintenance  and 
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operational  considerations.  These  analyses  will  be  used  to  update  the 
diagnostic  functional  allocation,  where  necessary,  to  resolve  ambiguities  or 
reduce  overlap.  These  analyses  shall  be  completed  by  the  Critical  Design 
Review  (CDR). 

o  Assessment  of  External  Diagnostics 

The  contractor  shall  deliver  at  the  Critical  Design  Review  detailed 
requirements  definition  for  external  test  equipment,  troubleshooting 
approaches  to  be  included  in  maintenance  manuals,  maintenance  aids,  and 
training  requirements.  These  requirements  shall  each  be  supported  by  a 
diagnostic  ambiguity  analysis  to  be  delivered  at  the  same  time.  The  analysis 
shall  describe  the  degree  to  which  diagnostic  ambiguities  are  reduced  and 
the  areas  where  there  is  redundancy  (overlap)  of  diagnostic  capabilities. 
The  analysis  shall  specifically  highlight  those  areas  where  the  combination  of 
embedded  and  external  diagnostics  cannot  unambiguously  detect  or  isolate 
a  fault  within  the  prescribed  diagnostic  time  limits  or  false  alarm  and  removal 
limits.  Its  results  will  be  used  to  update  requirements  for  external 
diagnostics. 

Diagnostic  Maturation  Program 

The  contractor  shall  establish  and  maintain  a  diagnostic  performance  data 
collection  system  and  conduct  diagnostic  performance  verification  tests  and 
demonstrations,  In  accordance  with  MIL-STD-470,  Task  301 ,  to  evaluate  the  effectiveness 
of  the  diagnostic  design.  As  a  minimum,  the  diagnostic  testing  should  include  the  insertion 
of  a  sufficient  sample  of  faults  to  provide  a  high  degree  of  confidence  in  the  results  of  the 
fault  coverage  demonstration/evaluation.  Figure  2  can  be  used  to  determine  the  95% 
confidence  limit  on  the  result  of  the  fault  coverage  demonstration.  In  addition,  diagnostic 
testing  should  include  system  operation  throughout  the  specified  environmental  range,  in 
order  to  evaluate  false  alarm  and  removal  deficiencies.  The  total  diagnostic  capability, 
both  embedded  and  external,  should  be  evaluated  concurrently  with  the  weapon  system 
itself.  This  is  true  for  maintainability  demonstrations  and  for  test  and  evaluations. 

Monitor  diagnostic  performance  whenever  the  system  is  operating  and  perform 
an  analysis  to  determine  whether  the  diagnostic  capabilities  are  operating  in  accordance 
with  the  design.  Based  on  the  maturation  results,  the  contractor  shall  take  corrective 
action  in  order  to  meet  diagnostic  capability  requirements.  The  contractor  shall  provide  a 
diagnostic  maturation  profile,  periodic  summary  of  diagnostic  performance  throughout  the 
development  cycle,  and  results  of  the  diagnostic  verification. 

The  diagnostic  performance  data  collection  system  shall  extend  through  Full- 
Scale  Development,  Production,  and  Deployment.  The  data  system  shall  be  designed  so 
that  the  performance  of  the  diagnostic  capability  can  be  ascertained  at  any  point  during 
the  acquisition,  production,  and  deployment  of  the  weapon  system.  It  shall  be  compatible 


1-22 


PREPARING  AN  RFP,  SOW,  SPECIFICATION 


REQUIREMENT  #1.2 


with  the  established  DoD  data  system,  which  will  be  employed  after  the  maintenance  of 
the  weapon  system  becomes  the  responsibility  of  the  government. 
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RGURE  2.  95%  CONROENCE  UMIT  FOR  DIAGNOSTIC  DEMONSTRATION 

The  contractor  shall  also  plan  for  the  transition  of  responsibility  to  the 
government  for  the  collection  and  analysis  of  diagnostics  data.  The  contractor  shall  make 
available  to  the  government  all  failure  analysis  and  trending  data  collected  as  a  result  of 
this  task.  The  data  shall  be  delivered  in  digital  form  in  a  format  acceptable  to  the 
government.  Field  data  collection  and  analysis  shall  be  automated  to  the  maximum  extent 
practical  and  cost  effective.  The  system  shall  be  integrated  to  the  maximum  extent 
practical  with  similar  data  collection  requirements  specified  elsewhere. 

Diagnostic  Capabiiity  Program  Pianning 

The  contractor  shall  develop  and  maintain  a  Diagnostic  Capability  Program  Plan 
which  describes  how  the  program  will  be  conducted.  The  program  plan  shall  be  included 
as  part  of  the  SEMP.  The  plan  describes  the  time  phasing  of  each  task  included  in  the 
contractual  requirements  and  its  relationship  to  other  tasks.  Diagnostic  issues  which 
relate  to  reliability,  maintainability,  logistics,  human  engineering,  safety,  etc.,  should  be 
addressed  in  those  individual  plans. 
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1 .  Identify  a  single  organizational  element  within  the  performing  activity  which 
has  overall  responsibility  and  authority  for  implementation  of  the  program. 
Establish  analyses  and  data  interfaces  among  the  organizational  elements 
responsible  for  each  of  the  elements  of  the  diagnostic  capability  and  other 
related  elements. 

2.  Develop  a  process  by  which  diagnostic  requirements  are  integrated  with 
other  design  requirements  and  disseminated  to  design  personnel  and 
subcontractors.  Establish  controls  for  assuring  that  each  subcontractor’s 
diagnostic  practices  are  consistent  with  overall  system  or  equipment 
requirements. 

3.  Identify  diagnostic  design  guides,  analysis  models  and  procedures  to  be 
imposed  upon  the  design  process.  Plan  for  the  review,  verification,  and 
utilization  of  diagnostic  data  submissions.  Explain  how  computer-aided 
design  tools  will  be  utilized  in  this  process. 


Diagnostic  Program  Reviews 


As  part  of  the  formal  reviews  (e.  g.,  PDR,  CDR)  which  are  conducted  during 
PSD,  the  design  and  performance  of  the  diagnostic  capability  shall  be  addressed.  These 
reviews  shall  be  coordinated  and  conducted  in  conjunction  with  reliability,  maintainability, 
human  engineering,  and  logistic  support  reviews,  whenever  possible.  Utilize  MIL-STD- 
1521  and  program  review  criteria  contained  In  MIL-STDs  470,  785,  2165  and  1388-1  as 
guidance. 


PDR/CDR 

o 

o 


RECOMMENDED  LIST  OF  DATA  DELIVERABLES 
AND  CANDIDATE  DIDs 

Embedded  Diagnostic  Design  Assessment  results  (DI-T-7199) 
External  Diagnostic  Capability  Detail  Specifications  (DI-E-3102) 


COMPLETION 

o  Diagnostic  Demonstration  results  (DI-R-7 1 1 3) 
o  Updated  Diagnostic  Capabilities  Field  Maturation  Plan  (DI-R-21597) 
o  Update  Diagnostics  Performance  Specification  (Dl  E-3103) 


These  candidate  DIDs  should  be  tailored  by  CDRLs. 
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PRODUCTION  PHASE 
(Sample  SOW) 


DIAGNOSTIC  MATURATION 

The  contractor  shall  mature  the  diagnostic  capability  in  accordance  with  the 
established  Maturation  Plan  to  assure  that  required  improvements  are  made  toward 
satisfying  the  updated  Diagnostics  Performance  Specification  at  each  maintenance  level. 
This  includes: 

a.  Maintaining  and  utilizing  the  diagnostic  data  system  to  measure 
performance  of  the  diagnostic  capability  and  take  required  corrective  action, 
in  accordance  with  the  incentive  and  warranty  provisions 

b.  Planning  and  transitioning  of  the  data  analyses  and  system  to  the 
government 

c.  Demonstrating  that  the  diagnostic  capability  satisfies  the  diagnostic 
requirements. 

RECOMMENDED  LIST  OF  DELIVERABLES 

0  External  Diagnostic  Demonstration  results  (DI-R-71 1 13) 

o  Maturation  results  (DI-R-71 05) 
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Preparing  a  Specification: 


REQUIREMENT  #1.2 


Preparation  of  the  diagnostic  portion  of  a  weapon  system  specification  is  a  job 
which  necessitates  a  full  understanding  of  the  design  and  fielding  of  the  diagnostic 
capability.  The  Government  Program  Manager  must  recognize  the  intricacies  of  this  job  to 
ensure  that  the  specifications  utilized  to  acquire  a  weapon  system  cleariy  define  the 
diagnostic  requirements. 

What  is  a  Failure? 

An  initial  requirement  in  the  specification  is  to  establish  the  definition  of  a  failure 
at  system,  subsystem,  and  unit  levels.  This  requirement  is  essential  in  demonstrating 
graceful  degradation  through  the  use  of  fault-tolerant  design,  reconfigurability, 
redundancy,  and  performance  monitoring.  A  failure  may  be  defined  as  causing  the 
mission  and  performance  requirements  of  the  prime  system  to  be  compromised. 

What  Means  are  Available  to  Perform  Diagnostics? 

Too  often  the  word  "diagnostic"  is  used  interchangeably  with  "test."  The 
specification  must  recognize  the  different  types  of  diagnostics,  which  include: 

o  Sensory  observations  (e.g.,  the  display  isn’t  on) 

o  Symptomatic  (e.g.,  usually  this  means  the  power  supply  isn’t  working) 

o  Test  (e.g.,  an  output  voltage  measure  of  zero) 

Means  through  which  systems’  diagnostics  can  be  addressed  include: 

o  Automatic  testing  (i.e.,  embedded  or  external) 

o  Manual  troubleshooting,  utilizing  technical  manuals,  troubleshooting 
procedures,  manual  test  equipment 

o  Operator  and  maintenance  technicians’  observations  and  various  forms  of 
performance  monitoring 

o  A  combination  of  the  above. 

What  Terms  Can  be  Used  in  Specifying  Diagnostic  Requirements? 

As  indicated  previously,  various  terms  may  be  used  to  specify  diagnostic 
requirements.  A  preferred  set  Is  contained  In  an  RADC  report,  "A  Rationale  and  Approach 
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for  Defining  and  Structuring  Testability  Requirements,"  (RADC-TR-85-150),  August  1985. 
The  set  includes  the  following: 

Fraction  of  Faults  Detected  (FFD) 

FFD  can  be  defined  as  that  fraction  of  failures  which  occur  over  operating 
time  which  can  be  correctly  identified  through  direct  observation  or  other  specified  means 
by  an  operator  and/or  other  specified  personnel  under  a  given  set  of  conditions.  The 
quantitative  definition  of  FFD  is: 


FFD -Ed 

Fa 

Where: 

F^  »  Number  of  actual  failures  (faults)  which  (will) 
occur  over  operating  time,  T. 

Fq  -  Number  of  actual  failures  correctly  identified 
through  direct  observation  and  other  specified 
means  by  an  operator  and/or  other  specified 
personnel  under  a  given  set(s)  of  conditions. 

In  specifying  FFD,  all  the  various  means  which  can  be  used  to  detect  faults  must 
be  taken  into  consideration.  The  requirement  for  FFD  should  be  stringent  enough  to 
exclude  the  application  of  the  types  of  detection  means  which  are 
unsatisfactory/unacceptable  for  the  system  needs/objectives/philosophies,  but  flexible 
enough  to  allow  the  contractor  to  tailor  his  design  cost  effectively.  In  general,  the  specific 
nature  and  mix  of  the  means  to  be  employed  to  achieve  a  given  minimum  FFD  should  be 
dependent  on  results  of  an  analysis  of  each  such  alternative  and  its  cost  and  performance 
effectiveness,  in  conjunction  with  other  system/equipment  design  factors  and 
requirements.  The  contractor  should  be  tasked  to  perform  such  analyses  and  provide 
results/recommendations  to  the  procuring  activity  based  on  these  factors. 

The  FFD  specification  parameter  must  be  specifically  defined  to  take  into 
account  frequency  of  failure  (failure  rates)  of  the  components  making  up  the  system,  it  is 
only  in  this  way  that  FFD  will  be  representative  of  what  occurs  during  operational  life. 

In  specifying  FFD,  care  must  be  taken  to  define  that  set  of  detection  conditions 
which  are  acceptable:  for  example,  who  can  perform  the  detection  function;  what  are  the 
acceptable  means  through  which  detection  can  be  performed;  during  which  equipment 
status  modes  can  detection  be  performed  (operation,  pre-  or  post-mission  checks,  etc.); 
and  whether  or  not  a  failure  must  be  detected  within  a  certain  period  of  time.  It  should 
also  be  noted  that  the  fraction  of  faults  detected  will  have  significant  effect  on  the  reliability 
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of  the  fault-tolerant  system  if  diagnostics  are  essential  part  of  the  system’s  failure/recovery 
mechanism. 

Fraction  of  Faults  Isolated  (FR) 

FFI  can  be  defined  as  that  fraction  of  failures  which  occur  over  operating  time 
which  can  be  correctiy  isolated  to  x  units,  or  fewer,  at  a  given  maintenance  echelon 
through  use  of  specified  means,  by  a  n.aintenance  technician  or  other  specified  personnel. 
The  quantitative  definition  for  FFI  is: 

FFI «  F, 


Where: 

Fy^  =  Number  of  actual  failures  (faults)  which  (will) 
occur  over  operating  time  T. 

F|  =  Number  of  actual  failures  (faults)  which  (will) 
occur  over  operating  time  T  that  can  be  correctly 
isolated  to  x  units,  or  fewer,  at  a  given  maintenance 
echelon  through  use  of  specified  diagnostic  scheme(s)/ 
procedure(s)  (or  a  defined  set  of  such),  by  a 
maintenance  technician  or  other  specified  personnel. 

In  specifying  FFI,  all  the  various  generic  means  acceptable  in  general  for  the 
mission/operational/maintenance  environment  which  can  be  used  to  isolate  faults  must  be 
taken  into  consideration.  The  requirement  for  FFI  should  be  stringent  enough  to  exclude 
the  application  of  isolation  means  which  are  known  in  general  to  be 
unsatisfactory/unacceptable  to  the  system  needs/maintenance  philosophy/objectives  but 
are  flexible  enough  to  allow  the  contractor  to  tailor  his  design  cost  effectively.  The  specific 
nature  and  mix  of  the  means  to  be  employed  should  be  dependent  on  the  results  of  an 
analysis  task  (levied  on,  and  performed  by,  the  contractor)  of  each  fault  isolation 
alternative,  in  conjunction  with  system/equipment  design  factors,  maintainability 
requirements,  and  support  system  needs.  Generally  speaking,  unless  there  is  clear 
evidence  that  unacceptable  weight,  volume,  or  cost  penalties  would  accrue,  primary 
diagnostic  means  based  on:  (1)  signal  tracing  and  analyses  through  the  use  of  schematics 
and  test  equipment,  and  (2)  repetitive  item  remove/replacement/performance  check 
actions  should  be  avoided. 

In  specifying  FFI  care  must  be  taken  to  indicate  the  conditions  under  which 
isolation  must  take  place: 

o  Where  it  takes  place  (i.e..  Organizational  Level,  Shop  Level) 
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o  What  are  the  acceptable  means  of  isolation  (i.e.,  built-in  test,  external 
testers,  general-purpose  testers,  peculiar  testers,  manual  means,  degree  of 
manual  means) 

o  Who  will  perform  the  isolation  (i.e.,  operator  or  maintenance  technician) 

0  Its  constraints  (i.e.,  prohibition  of  wholesale  removal  of  units,  time  allowable) 

o  Its  second  isolation  tier  requirements  (what  happens  after  isolation  to  proper 
ambiguity  level) 

o  The  time  constreints  levied  by  the  maintainability  requirement. 

The  FFI  parameter  must  be  specifically  defined  to  take  into  account  frequency  of 
failure  (failure  rates)  of  the  components  making  up  the  system.  It  is  only  in  this  way  that 
FFI  will  be  representative  of  what  occurs  during  operational  life. 

False  Alarm  Rate 

A  false  alarm  Is  defined  as  an  apparent  indication  of  failure  when,  in  fact,  no 
failure  exists.  The  false  alarm  rate  is  the  number  of  false  alarms  per  unit  of  time. 

Intermittent  faults  can  be  difficult  to  distinguish  from  false  alarms  during 
operational  test  and  in  use.  A  properly  structured  qualification  test,  however,  can  exclude 
the  influence  of  intermittent  faults. 

False  alarm  rates  are  controllable  through  the  use  of  such  design  techniques 
and  features  as: 

o  Scope  and  magnitude  of  performance  monitoring 

o  Definition  of  test  tolerances 

o  Transient  monitoring  and  control 

o  Multiple-run  decision  logic 

0  Environmental  effects  filtering  and  identification. 

Fraction  of  Erroneous  Fault  Isolation  Results  (FER) 

FEFI  is  the  fraction  of  BIT  or  external  tester  isolations  that  identify  the  wrong 
removable  unit  (subunit)  or  group  of  units  (subunits)  as  failed.  FEFI  is  primarily  a  design 
problem  resulting  either  from  test  system  design  error  or  low  sensitivity  thresholds  and 
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tolerance  levels  of  system/equipment  components  and/or  signals.  It  can  have  serious 
consequences  by  creating  confusion  during  fault  isolation  and  by  eroding  maintenance 
technician  confidence  in  the  test  system.  The  quantitative  definition  is: 

FEFI  =  Fg 

Where; 

F^  =  Number  of  actual  failures  (faults)  which  (will) 
occur  over  operating  time  T. 

Fg  *x  Number  of  actual  failures  (faults)  which  (will) 
occur  over  time  T  that  are  isolated  to  a 
nonfalled  unit  or  group  of  units. 

What  Does  100%  Fault  Detection/Fault  Isolation  Mean? 

In  defining  FFD  as  a  contractual  requirement  for  most  programs,  it  is  sometimes 
simpler  to  exclude  those  types  of  direct  detection  means  (for  example,  detection  through 
the  use  of  technicians)  which  would,  In  general,  be  unsatisfactory  to  a  given  mission 
environment  than  to  define  those  that  are  acceptable.  The  fact  that  an  FFD  requirement  is 
imposed  should  not  imply  that  100%  of  ail  expected  failures  should  not  be  detectable.  The 
contractor  should  be  tasked  with  the  development  of  cost-effective,  defined  procedures  to 
detect  all  expected  failures.  All  of  these,  however,  need  not  be  direct  means  or  belong  to 
the  type  of  direct  means  which  are  defined  as  satisfactory  for  general  mission  operational 
use,  provided  maintainability  and  other  requirements  can  still  be  met.  Detection  can 
include  direct  or  indirect  indications  to  an  operator,  the  use  of  maintenance  technicians  or 
other  personnel  performing  in  accordance  with  a  series  of  defined  routines,  or  some 
combination  of  these. 

For  FFI  100%  coverage  is  required,  which  simply  states  that  using  a 
combination  of  all  diagnostic  resources,  all  faults  can  be  isolated,  given  an  adequate 
amount  of  time.  Applying  restrictions  in  time  means  that  100%  of  all  expected  faults  will  be 
isolatable,  but  a  certain  fraction  (1-FFI)  may  have  ambiguity  levels  greater  than  the  value 
stated  or  be  isolatable  through  means  which  are  definable,  but  which  do  not  belong  to  the 
class  of  diagnostic  means  cited  as  being  acceptable  for  general  use  in  the  given  mission 
or  use  environment.  Consideration  must  be  given  as  to  how  and  where  isolation  to  the 
faulty  unit(s)  must  take  place. 

In  summary,  specifications  should  indicate  a  1 00%  fault  detection/fault  isolation 
coverage  at  each  maintenance  level  (e.g.,  combinations  of  automatic  and  manual 
troubleshooting  means  should  equal  100%).  This  does  not  mean  that  100%  of  faults  can 
be  isolated  to  a  given  unit  within  a  given  time  using  specific  diagnostic  resources. 
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What  is  Diagnostic  Growth? 

Another  aspect  which  is  recommended  to  be  introduced  is  that  of  diagnostic 
growth-similar  in  concept  to  the  already  established  reliability  growth.  This  growth 
requirement  is  especially  important  in  the  maturation  of  the  system.  Figure  3  is  a 
conceptual  version  of  this  growth  process.  Demonstrations  that  these  goals,  or 
requirements,  have  been  achieved  at  various  phases  of  weapon  system  development 
must  be  tailored  to  the  specific  weapon  system  acquisition  strategy.  For  instance,  if  the 
performance  of  an  aircraft  is  to  be  evaluated  at  the  conclusion  of  DemA/al,  then  the  entire 
diagnostic  capability  for  the  aircraft  should  reach  the  specified  requirement  at  that  point  in 
time.  On  the  other  hand,  if  only  specific  units  (usually  high  risk)  of  a  weapon  system  are 
developed  during  Oem/Val,  then  the  diagnostic  capability  for  only  those  specific  units  may 
be  demonstrated.  The  maturation  of  a  diagnostic  capability  for  the  entire  weapon  system, 
in  most  cases,  will  extend  into  the  Deployment  Phase.  Thus  the  required  fault 
detection/fault  isolation  capability  usually  cannot  be  achieved  until  that  time. 


GOAL  GOAL  GOAL  GOAL 


RGURE  3.  DIAGNOSTIC  GROWTH  CONCEPT 
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The  diagnostic  portions  of  the  weapon  system  specification  differ,  depending  on 
the  stage  of  development.  Normally,  these  specifications  take  the  form  of  a  Preliminary 
System  Specification,  resulting  from  Concept  Exploration;  a  System  Specification,  derived 
from  the  Demonstration  and  Validation  Phase;  and  a  Configuration  item  Development 
Specification,  which  allocates  requirements  down  to  subsystem  and  item  levels  (A  more 
complete  definition  of  the  various  types  of  specifications  is  contained  in  MiL-STD-490, 
Specification  Practices^.  The  following  examples  of  diagnostic  portions  of  specifications 
follow  this  form.  The  content  must  be  tailored  to  fit  a  specific  system  requirement. 

Preliminary  System  Specification 

The  Preliminary  System  Specification  is  a  result  of  Concept  Exploration  Phase 
studies,  prior  to  conducting  the  detailed  diagnostic/testability  requirements  analysis  during 
the  Demonstration  and  Validation  Phase. 

Quantitative  testability/diagnostic  parameters  are  not  specified  in  the  Preliminary 
System  Specification.  Rather,  qualitative  system-level  diagnostic/testability  goals  are 
included. 

The  model  paragraphs  below  may  be  included  in  the  Preliminary  System 
Specification  primarily  to  alert  the  performing  activity  that  diagnostic/testability  is 
considered  to  be  an  important  aspect  of  the  design  and  that  quantitative  requirements  will 
be  imposed  in  the  final  System  Specification. 


3.X.X  Design  for  Testability 

3.XJ(.f  Partitioning.  The  system  .shaO  be  partitioned  based,  in  part,  upon 
the  ability  to  confldently  boiate  fauib. 

3.X.X.2  Test  Poinb.  Each  unit  within  the  system  shall  have  suiTIcient  test 
polnb  for  the  measurement  or  stimulus  of  internal  circuit  nodes  so  as  to 
achieve  an  inherently  high  level  of  fault  detection  and  Isolation. 

3.X.X.3  Built-In  Test.  Mission  critical  fhnetions  shall  be  monitored  by 
Built-in-Test.  BIT  tolerances  shall  be  set  to  optimize  fault 
detection/fabe  alarm  characteristics.  BIT  Indicators  shall  be  designed 
for  maximum  utilization  by  intended  personnel  (operator/mahiblner). 


System  Specification 

Quantitative  testability/diagnostic  requirements  are  derived  from  the 
tradeoff  anaiysis  during  the  Demonstration  and  Vaiidation  Phase  and  are 
incorporated  in  the  System  Specification.  Requirements  may  be  expressed  in  terms 
of  goals  and  thresholds  rather  than  as  a  single  number.  Requirements  for 
diagnoslic/testabiiity  in  a  System  Specification  are  provided  in  the  foilowing  modei. 
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3.2.4. x  DIagnostIc/Teatablllty-ThB _ syatam  (Insert  name)  shall  be  designed  for 

testability  and  constructed  to  permit  the  status  of  the  system  and  the  unambiguous  location 
of  faults  to  be  confidently  determined  and  reported  In  a  timely  fashion. 

3.2.4. X.1  Partitioning  (functional  Modularity)  •  Syatem/Subayatema  will  be  partitioned  Into 
Line  Replaceable  Units  (LRU)  based  on  the  function,  minimum  or  optimum  number  of 
Interconnections,  the  ability  to  fault  Isolate  to  the  correct  unit  LRUs  will  be  subdivided  Into 
next  level  replaceable  Items  (e.g..  Shop  Replaceable  units,  or  SRU)  based  on  function, 
minimum  or  optimum  number  of  Interconnections,  and  the  ability  of  personnel  with  the  aid  of 
support  equipment,  training,  and  technical  manuals  to  fault  Isolate. 

3.2.4. X.2  Teat  Points  and  Contacts  -  Test  points  and  contacts  shall  be  conveniently  located 
and  have  safe  access  to  signal  nodea  and  shall  be  provided  for  the  measure  or  Injection  to 
significant  parameters  for  the  purpose  of  evaluating  or  troubleshooting  the  circuit 
mechanisms,  the  number  and  choice  of  accessible  nodes  shall  be  sufficient  to  obtain  the 
equipment  fault  detectlon/laolatlon  requirements  listed  herein. 

3.2.4. X.3  Diagnostic  Capability  -  For  each  level  of  maintenance:  all  diagnostic  resources  shall 
be  Integrated  to  provide  a  consistent  and  complete  diagnostic  capability.  A  complete 
diagnostic  capability  must  Identify  the  diagnostic  resources  that  will  be  used  to  have  full 
FD/Fl  coverage.  The  degree  of  diagnostic  automation  shall  be  consistent  with  the  proposed 
personnel  sMIl  levels  and  maintenance  repair  dmes. 

3.2.4. X.4  Built-In  Teat  •  Built  In  Test  (BIT)  provisions  shall  be  designed  Into  the 

_ system  to  teat  ayatem/equipment  and  to  Inform  the  operator  of  the  ability  of 

the  equipment  to  perform  a  particular  mission. 

3.2.4. X.4.1  On-Line  BIT  Performance  Monitoring  -  The  on-line  BIT  performance  monitoring 
features  shall  be  operative  and  shall  provide  valid  performance  Indlcatlona  prior  to  and  during 
operation.  The  performance  monitoring  operation  shall  be  automatic  and  continuous  and 
shall: 

1.  Ensure  aubayatema  are  operational  and  are  capable  of 
satisfying  their  designated  mission  functions. 

2.  Detect  any  system  failure  or  degradation  which  would  adversely 
affect  the  system’s  ability  to  satisfy  Its  mission  objectives. 

All  BIT  Implementations  of  this  requirement  shall  be  contained  within  the  system 
or  subsystem  hardware  and  will  not  degrade  mission  performance  at  any  time. 

3.2.4.X.4. 1. 1  No-Qo  Condition  Detection  -  The  system  on-line  BIT  performance 

monitoring  features  shall  detect  at  least _ percent  of  all  no-go  condition 

occurrences  over  the  mission  time.  (As  applied  at  the  weapon  system  level  or  as 
applied  Independently  at  the  subsystem  level.) 


Continued  on  the  following  page... 
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3.2.4. X.4.1.2  False  Alarms  •  The  number  of  false  alarms  shall  not  exceed 

_ percent  of  all  Indicated  no-go  condition  occurrences  or  alternately  no 

more  than _ Indicated  no-go  condition  occurrences  In  any  24-hour  period  of 

system  operating  time. 

3.2.4. X.4.1.3  Performance  Monitor  and  Self-Teat  Data  -  Performance  monitor  and 
sMf-test  data  shall  be  transmitted  In  a  manner  such  that  the  transmitted  data  shall 
follow  the  actual  condition  of  the  system,  that  la,  a  malfunction  which  corrects 
Itself  shall  change  the  faPIt  data  line  accordingly. 

3.2.4. X.4.2  Off-Line  BIT  -  The _ system  BIT  provision  shall  furnish  the 

means  for  an  operator  to  Initiate  BIT  testa  tor  purposes  of  determining  and 
displaying  the  functional  status  of  the  ayatema/aubayatema  Including  a  fault 
detection  and  Isolation  capability.  The  Intended  use  of  the  off-line  BIT  testa  la  two¬ 
fold:  as  a  System  readiness  teat  to  permit  operating  crew  to  accumulate  status 
and  fault  Information  on  an  opportunity  basts  prior  to  and  during  operations;  and 
the  verify  a  fault  Indicated  during  operation  and  to  Isolate  the  fault  at  the 
Organizational  level  of  maintenance. 

3.2.4. X.4.2.1  No-Go  Condition  detection  -  The  system  off-line  BIT  features  shall 

detect  at  lease _ percent  of  all  no=go  condition  occurrences,  (as  applied  at 

the  weapon  system  level  or  as  applied  Independently  at  the  subsystem  level.) 

3.2.4. X.4.2.2  False  Alarms  -  The  number  of  false  alarms  shall  not  exceed _ 

percent  of  all  Indicated  no-go  conditions  occurrences  or  alternately  no  more  than 

_ Indicated  no-go  condition  occurrences  In  any  24-hour  period  of  system 

operating  time. 

3.Z4.X.4.Z3  Off-Line  BIT  Fault  detection  -  The  off-line  BIT  fault  detection  capability 
shall  be  designed  to  monitor,  detect,  and  evaluated  faults  on  all  system  or 
subsystem  functions  available  at  the  system  or  subsystem  Interface.  When  a  fault 
or  system  degradation  la  detected,  the  off-line  BIT  provision  may  determine  the 
amount  of  degradation  and  automatically  branch  Into  the  appropriate  diagnostic 
fault  Isolation  routine. 

3.2.4. X.4.2.4  Off-Line  BIT  Fault  Isolation  -  The  off-line  BIT  fault  Isolation  routines 
shall  be  provided  at  each  fault  detection  point  and  shall  be  automatically  entered 
when  a  no-go  la  detected.  The  off-line  BIT  shall  provide  fault  Isolation  to  one  Line 

Replaceable  Unit _ percent  of  the  time,  fault  Isolation  to _ or  fewer 

Line  Replaceable  Units _ percent  of  the  time.  In  no  case  shall  the  ambiguity 

group  be  greater  than _ LRU. 


Continued  on  the  following  page.. 


3.2.4. X.2.5  Off-Line  BIT  Fault  Isolation  Tima  •  The  off-line  BIT  Fault  Isolation  time 
shall  be  consistent  with  the  nquirements  of  the  Organizational  Level  Mean  Time  to 
repair  (MTTR)  requirements. 

3.2.4. X.4.3  BIT  Self  Test  -  BIT  self  test  provisions  shall  be  Incorporated  Into  the 

_ system.  The  time  for  the  BIT  self  teat  shall  be  leas  than _ (and/or 

the  duty  cycle  of  the  BIT  self  teat  shall  be _ ).  [The  BIT  failure  rate  shall  be 

leas  than _ percent  of  the  prime  system  BIT  failure  Indication  rate.] 

3.2.4. X.4.4  Fall  Safe  Provisions  -  The  circuits  and  devices  which  provide  BIT  and 
fault  Isolation  functions  shall  be  designated  In  such  a  manner  that  failure  of  these 
circuits  and/or  devices  will  not  cause  a  critical  failure  or  unsafe  action  of  the 
system. 

3.2.4. X.4.S  Skill  Levels  -  A  personnel  skill  level  of _ la  required  to  permit  the 

accomplishment  of  all  actions  associated  with  the  fault  Isolation  and 
removal/replacement  of  LRUs  at  the  operatlonal/O-level.  BIT  provisions  and, 
where  required,  0-level  teat  equipment  and  maintenance  procedures  will  be  used 
to  provide  fault  Isolation  within  the  MTTR  specification. 

3.2.4. X.S  Teat  Equipment  Interface  -  Signals  shall  be  Included  at  the  module 
Interface  which  maximizes  the  similarity  of  bullt-ln-teating  by  the  equipment  and 
external  testing  by  manual  teat  equipment  and/or  on  ATE  systems.  The  system 
shall  be  designed  for  compatibility  for  teat  with  the  selected  or  targeted  ATE  (or 

_ (Insert  teat  equipment  name/dealgnator)).  Maximum  use  shall  be  made  of 

operational  pins  to  provide  teat  control  and  access  to  satisfy  the  fault 
detectlon/fault  Isolation  requirements  of  external  test 

3.2.4. X.6  Teat  Tolerances  -  Appropriate  tolerances  and  signal  limits  shall  be 
established  In  diagnostic  roudnea  at  each  level  which  the  ayatem/equipmenta  are 
aubfect  to  testing  such  that  false  alarms  and  Retest  Okay  rates  are  minimized. 

3.2.4. X.7  Technical  Information  Access  Time  -  Average  time  required  for  the 

maintenance  technician  to  access  maintenance  technical  Information  shall  be  less 
than _ minutes  at  the  organizational  level. 


Configuration  item  Deveiopment  Specification 

A  model  testability  specification  suitable  for  inclusion  in  the  Cl 
development  specification  is  provided  as  follows; 
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3.2.4. x  Testability /Diagnostic  -  The _ subsystem/item  (insert  name)  shall  be 

designed  for  testability  and  constructed  to  permit  the  status  of  the  subsystem/item  and 
the  unambiguous  location  of  faults  to  be  confidently  determined  and  reported  in  a  timely 
fashion. 

3.2.4. x. 1  Partitioning  (Functional  Modularity)  -  Subsystems/Items  will  be  partitioned  into 
Shop  Replaceable  Units  (SRU)  based  on  the  function,  minimum  or  optimum  number  of 
interconnections,  the  ability  to  fault  isolate  the  correct  unit  and  the  ability  of  personnel 
with  the  aid  of  support  equipment,  training,  and  technical  manuals  to  fault  isqlate  to  the 
correct  unit. 

3.2.4. X.2  Test  Points  and  Contacts  -  Test  points  and  contacts  shall  be  conveniently 
located  and  have  safe  access  to  signal  nodes  on  the  unit  under  test,  and  shall  be 
provided  for  the  measure  or  injection  of  significant  parameters  for  the  purpose  of 
evaluating  or  troubleshooting  the  circuit  mechanisms.  The  number  and  choice  of 
accessible  nodes  shall  be  sufficient  to  obtain  the  equipment  fault  detection/isolation 
requirements  listed  herein. 

3. 2.4. x. 3  Diagnostic  Capability  -  For  each  level  of  maintenance:  all  diagnostic 
resources  shall  be  integrated  to  provide  a  consistent  and  complete  diagnostic  capability. 
A  complete  diagnostic  capability  must  identify  the  diagnostic  resources  that  will  be  used 
to  have  full  FD/FI  coverage.  The  degree  of  diagnostic  automation  shall  be  consistent 
with  the  proposed  personnel  skill  levels  and  maintenance  repair  items. 

3.2.4. X.4  Built-In  Test  •  Built-In  Test  (BIT)  provisions  shall  be  added  to  the _ 

subsystem/item  to  satisfy  system  level  performance  monitoring  and  off-line  BIT 
requirements. 

3.2.4. X.4.1  On-Line  BIT  Performance  Monitoring  -  The  on-line  BIT  performance 
monitoring  features  shall  be  operative  and  shsUI  provide  valid  performance  indications 
prior  to  and  during  operation.  The  performance  monitoring  operation  shall  be  automatic 
and  continuous  shall  be  automatic  and  continuous  and  will  monitor  self-contained  signal 
generating  circuitry.  All  BIT  implementations  of  this  requirement  shall  be  contained 
within  the  system  or  subsystem  hardware  and  wilt  not  degrade  mission  performance  at 
any  time. 

3. 2. 4. x. 4. 1.1  No-Qo  Condition  Detection  -  The  system  on-line  BIT  performance 

monitoring  features  shall  detect  at  least _ percent  of  all  no-go  condition 

occurrences.  (As  applied  independently  at  the  subsystem  level.) 

3. 2. 4. x. 4. 1.2  False  Alarms  -  the  number  of  false  alarms  shall  not  exceed _ 

percent  of  all  indicated  no-go  condition  occurrences  or  alternately  no  more  than _ 

indicated  no-go  condition  occurrences  in  any  24-hour  period  of  system  operating  time. 

3.2.4. X.4.1 .3  Performance  Monitor  and  Self-Test  Data  -  Performance  monitor  and  self¬ 
test  data  shall  be  transmitted  in  a  manner  such  that  the  transmitted  data  flow  shall  be 
transmitted  in  a  manner  such  that  the  transmitted  data  shall  follow  the  actual  condition  of 
the  system,  that  is,  a  malfunction  which  corrects  itself  shall  change  the  fault  data  line 
accordingly. 


Continued  on  the  following  page... 
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3.2.4. X.4.2  Off-Line  BIT-  The _ subsyetem  BIT  provision  shall  furnish  the 

means  for  an  operator  to  Initiate  BIT  testa  at  the  system  level  tor  purpose  of 
determining  and  displaying  the  functional  status  of  the  ayatem/aubaystema 
Including  a  fault  detection  and  Isolation  capability.  The  Intended  use  of  the  off-line 
BIT  teat  la  two-fold:  aa  a  System  readiness  Teat  to  permit  operating  crew  to 
accumulate  status  and  fault  Information  an  opportunity  basis  prior  to  and  during 
operations;  and  to  verify  a  fault  Indicated  during  operation  and  to  Isolate  the  fault 
at  the  O-level  of  maintenance. 

3.2.4. X.4.2.1  No-Go  Condition  Detection  -  The  system  off-line  BIT  features  shall 

detect  at  least _ percent  of  all  no-go  condition  occurrences.  (Aa  applied 

Independently  at  the  subsystem  level.) 

3.2.4. X.4.2.2  False  Alarms  -  The  number  of  false  alarms  shall  not  exceed _ 

percent  of  all  Indicated  no-go  condition  occurrences  or  alternately  no  more  than 

_ Indicated  no-go-conditlon  occurrences  In  any  24-hour  period  of  system 

operating  time. 

3.2.4. X.4.2.3  Off-Line  BIT  Fault  Detection  -  The  off-line  BIT  fault  detection 
capability  shall  be  designed  to  monitor,  detecL  and  evaluate  faults  on  all  system  or 
subsystem  functions  available  at  the  system  or  subsystem  Interface.  Vlfhen  a  fault 
or  system  function  degradation  la  detected,  the  off-line  BIT  provisions  shall 
determine  the  amount  of  degradation  and  automatically  branch  Into  the 
appropriate  diagnostic  fault  Isolation  routine. 

3.Z4.X.4.2.4  Off-Line  BIT  Fault  Isolation  •  The  off-line  BIT  fault  Isolation  routines 
shell  be  provided  at  each  fault  detection  decision  point  and  shall  be  automatically 
entered  when  a  no-go  la  detected.  The  off-line  BIT  shall  provide  fault  Isolation  to 

one  Shop  Replaceable  Unit _ %  of  the  time,  fault  Isolation  to _ or  fewer 

Shop  replaceable  Units _ %  of  the  time. 

3.Z4.X.4.2.5  Off-Line  BIT  Fault  Isolation  Time  -  The  off-line  BIT  fault  Isolation  time 
shall  be  consistent  with  '?qii:r^ments  of  Mean  Time  To  RePaIr  (MTTR) 
requirements. 

3.2.4. X.4.3  BIT  Self  Teat  -  BIT  self  teat  provisions  shall  be  Incorporated  Into  the 

_ aubsystem/ltem.  The  time  tor  the  BIT  self  teat  shall  be  leas  than _ 

(and/or  the  duty  cycle  of  the  BIT  self  teat  shall  be _ ).  The  BIT  failure  rate  shall 

be  leas  than _ %  of  the  prime  system  BIT  failure  Indication  rate. 

3.2.4. X.4.4  Fall  Safe  Provisions  -  The  circuits  and  devices  which  provide  BIT  and 
fault  Isolation  functions  shall  be  designed  In  such  a  manner  that  failure  of  these 
circuits  and/or  devices  will  not  cause  a  critical  failure  or  unsafe  action  of  the 
aubsystem/ltem. 


Continued  on  the  following  page.. 


3.2.4. X.6  Skill  Levels  -  A  personnel  skill  level  of _ Is  required  to  permit  the 

sccompllshment  of  sll  actions  associated  with  the  fault  Isolation  and 
removal/replacement  of  SSUa  at  the  Intermediate  maintenance  level.  BIT 
provisions,  test  equipment  and  maintenance  procedures  will  be  used  to  provide 
fault  Isolation  wlUtIn  the  M7T/7  specification. 

3.2.4. X.7  Teat  Equipment  Interface  -  Signals  shall  be  Included  at  the  module 
Interfaces  which  maximize  the  similarity  of  bullt-ln  testing  by  the  equipment  and 
off-board  testing  by  manual  teat  equipment  and/or  on  ATE  systems.  The  system 
shall  be  designed  for  compatibility  for  teat  with  target  off-line  automatic  teat 
equipment  maximum  use  shall  be  made  of  operational  pins  to  provide  teat  control 
and  access  to  satisfy  the  fault  detectlon/fault  Isolation  requirements  of  off-board 
teat 

3.2.4. X.8  LRU  Fault  Detecdon/laolatlon  Requirements  -  The  following  requirements 
apply  to  fault  detectlon/laolatlon  capability  at  the  Intermediate  level  of 
maintenance  using  automatic  teat  resources  (ATE/TPS  and  BIT). 

-  Fault  Isolation  shall  be _ percent  of  all  organizational 

detected  failures. 

•  Average  (or  maximum)  teat  time  for  GO/NO-GO  end-to-end  testa 
shell  be  leas  than _ (mlnutea/houra). 

-  Maximum  rate  of  false  NO-GO  Indications  resulting  In  Cannot 

Duplicates  and  retest  Okays  shall  be _ percent  of  all 

Organizational  level  detected  failures. 

-  Fault  Isolation  capability  shall  provide  fault  Isolation  to  one  SRU 

_ percent  of  the  time,  fault  Isolation  to _ or  fewer  SRUa 

_ percent  of  the  time.  In  no  case  shall  the  ambiguity  group 

size  be  greater  than _ SRU. 

-  A  verage  (or  maximum)  diagnostic  fault  Isolation  time  shall  be  less 

than _ (mlnutea/houra). 

3.2.4. X.9  SRU  Fault  Detectlon/laolatlon  Requirement  -  The  following  requirements 
apply  to  fault  detectlon/laolatlon  capability  at  the  Depot  level  of  maintenance  using 
automatic  test  resources  (ATE/TPS  and  BIT). 

-  Fault  Isolation  shall  be _ percent  of  all  detected  failures. 

-  Average  (or  maximum)  teat  time  for  GO/NO-GO  end-to-end  testa 

shall  be  leas  than _ (mlnutea/houra). 

-  Maximum  rate  of  false  NO-GO  Indications  resulting  In  Retest 

Okays  shall  be _ percent  of  all  detected  failures. 

-  Fault  Isolation  capability  shall  provide  fault  Isolation  to  one 

component _ percent  of  the  time,  fault  Isolation  to _ or 

fewer  components _ percent  of  the  time.  In  no  case  shall  the 

ambiguity  group  size  be  greater  than _ components. 

-  Average  (maximum)  diagnostic  fault  Isolation  teat  time  shall  be 

less  than _ (mlnutea/houra). 


PREPARING  AN  RFP  ,SOW,  SPECIRCATION 


REQUIREMENT  #1.2 


CHECKLIST 

of  Does  the  RFP/SOW  require  the  integration 

of  oil  of  the  diagnostic  elements  os  port  of  the 
system  engineering  and  design  process? 

E2f  Is  there  use  of  automation  in  the  diagnostic 
design  process? 

Is  the  contractor  asked  to  propose  approaches 
for  diagnostic  design  verification 
prior  to  building  hardware? 

E2f  is  the  diagnostic  specification  constant  with,  and 
supportive  of,  the  MTTR  requirements? 


requirements? 


Do  the  Evaluation  Factors  for  Award  provide 
information  for  the  bidder  to  stress  the 
importance  of  addressing  the  integration  of 
testability/diagnostic  issues  and  requirements? 

E2f  Does  the  SOW  stress: 

—  Use  of  the  SEMP  as  the  principal  diagnostic 
planning  document? 

—  Preparation  of  a  Maturation  Program  Plan? 

Has  the  sample  SOW  been  tailored  for  use? 

Is  It  required  that  all  diagnostic  elements  be 
evaluated  as  a  single  diagnostic  capability  during 
maintainability  demonstrations?  OT&E? 


ions? 


E2f  Does  the  specification  address  necessary  FD/FI 
requirements  at  each  level  of  maintenance? 

E2^  Has  the  concept  of  "diagnostic  growth"  been 
Invoked  in  the  diagnostic  specification? 
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WEAPON 

SYSTEM 

ACQ.  PHASE 

OPER. 

REQMTS. 

CONCEPT 

EXPLOR. 

DEM/ 

VAL 

FSD 

PROD/ 

DEPL 

WEAPON 

SYSTEM 

ACTIVITIES 

zs  zs^  zx 

CONTRACT  AWARD 

DIAGNOSTIC 

ACTIVITIES 

zra  zra  AA  A 

G  C  G  C  G  C  C 

PREPARE/UPDATE  DIAGNOSTIC  CAPABILITY  PROGRAM  PLANS 

C-  CONTRACTOR  PREPARED 
G-  GOVERNMENT  PREPARED 


DIAGNOSTIC  ACTIVITY 


Program  planning  is  required  to  ensure  that  the  development  and  support  of  the 
diagnostic  capability  is  properly  managed  throughout  the  acquisition  of  a  weapon  system. 
This  planning  must  address  how  this  development  will  be  conducted  to  achieve  this  goal. 

PROCEDURE 

Program  planning  for  the  development  of  the  diagnostic  capability  is  required 
throughout  the  acquisition  of  the  weapon  system.  This  planning  is  the  responsibility  of 
both  the  Government  and  the  Contractor’s  Program  Manager.  Although  Government 
Program  Managers’  planning  requirements  differ  from  military  service  to  military  service  (e. 
g.,  Program  Management  Plan,  Acquisition  Strategy),  the  objective  remains  the  same-to 
assure  that  the  program  management  functions  are  properly  planned  and  executed  by  the 
government. 

On  the  other  hand,  the  contractor’s  program  planning  is  aimed  at  ensuring  that 
the  development  of  the  diagnostic  capability  Is  properly  planned  and  executed.  This 
program  planning  can  take  the  form  of  a  single  Diagnostic  Capability  Program  Plan  or  it 
can  bo  incorporated  in  a  series  of  program  plans  which  are  described  in  a  number  of 
programmatic-type  military  standards.  The  requirement  for  these  planning  documents 
needs  to  be  defined  in  the  RFP/contract  Statement  of  Work.  To  avoid  unnecessary 
duplication  of  program  plans,  the  inclusion  of  this  planning  information  in  existing 
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documents  is  preferred.  The  Government  Program  Manager’s  responsibility  is  to  review 
these  plans  for  adequacy. 

GUIDANCE 


As  cited  above,  the  Government  Program  Manager’s  planning  documents  can 
take  a  variety  of  forms,  ranging  from  a  Program  Management  Plan  to  an  Acquisition 
Strategy  Document.  The  format  varies,  as  does  the  content.  However,  in  preparing  these 
program  plans,  several  diagnostic  issues  must  be  addressed.  These  are: 

1.  The  establishment  of  a  central  focal  point  within  the  program  office 
responsible  for  meeting  the  requirements  for  the  entire  diagnostic  capability 

2.  The  establishment  of  means  for  interfacing  with  other  organizations, 
including: 

o  The  training  community 
o  The  user  community 
0  The  logistics  support  community. 

3.  Moans  for  integrating  the  various  design  and  support  functions  which  affect 
the  diagnostic  capability,  including  the  integration  of  the  various  "ility" 
functions 

4.  Means  for  assuring  that  diagnostic  requirements  for  GFE/CFE  are  satisfied 

5.  Means  to  assure  that  government  activities,  such  as  preparation  of  an  RFP 
and  conducting  technical  reviews,  address  diagnostic  needs. 

For  those  management-type  plans  that  are  prepared  by  the  contractor,  the 
Government  Program  Manager’s  responsibility  is  to  review  these  plans  for  adequacy. 
Guidance  on  the  requirements  for  these  plans  is  contained  in  Requirement  1 .2,  Preparing 
an  RFP/SOW/Specification. 
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REQUIREMENT  #1.3 


CHECKLIST 

GOVERNMENT-PREPARED  PLANNING  DOCUMENTS 

of  Has  a  person  within  the  program  office  been  assigned 
to  be  solely  responsible  for  the  adequacy  of  the 
diagnostic  capability? 

Have  adequate  interfaces  been  established  with 
training,  user,  and  support  communities? 

Does  the  plan  adequately  address  the  integration 
of  the  diagnostic  capability? 

CONTRACTOR-PREPARED  PLANNING  DOCUMENTS 

E2f  Does  the  SEMP  address  design  of  the  diagnostic  cap¬ 
abilities  as  an  integral  part  of  the  system  engineering 

§rocess  (i.e.,  emphasis  on  Parts  I  and  II  of  tne 
EMP,  as  opposed  to  Part  III)? 

EZ^  Does  Part  III  of  the  SEMP  require  combined  "llity" 
and  logistic  demonstrations? 

E2f  Are  the  diagnostic  implications  included  in  the 
various  "ility"  and  logistic  plans? 


PREPARATION  OF  SCPs/DCPs 


REQUIREMENT  #1.4 


WEAPON 

SYSTEM 

ACQ.  PHASE 

OPER. 

REQMTS. 

CONCEPT 

EXPLOR. 

DEM/ 
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FSD 

PROD/ 

DEPL 

WEAPON 

SYSTEM 

ACTIVITIES 

Z 

's. 

\  1 

AP^ 

PRO 

k  Z 
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CEED 

i  A 

LOGISTIC 

REVIEW 

DIAGNOSTIC 

ACTIVITIES 

ZX  ZX  AAA 

SCP  DCP  DCP 

UPDATE 

DIAGNC  .lie  ACTIVITY 

Prior  to  Milestones  I  through  V  (DoD  Instruction  5000.2),  the  preparation  of  a 
paper  is  required  to  summarize  the  results  of  the  acquisition  and  deployment  of  a  major 
weapon  system.  Prior  to  Dem/Vai,  a  System  Concept  Paper  (SCP)  is  required.  Prior  to 
PSD,  preparation  of  a  Decision  Coordinating  Paper  (DCP)  is  required.  An  update  of  the 
DCP  is  required  prior  to  Miiestones  III,  IV,  and  V.  The  SCP  and  the  DCPs  for  Milestones  II 
and  III  are  required  to  secure  approval  to  proceed  to  the  next  acquisition  phase.  Milestone 
IV  is  a  logistic  readiness  and  support  review,  which  is  conducted  one  or  two  years  after 
deployment  to  assure  that  operational  readiness  and  support  objectives  are  achieved. 
Milestone  V  is  a  major  upgrade  or  system  replacement  decision,  which  also  requires  an 
updating  of  the  DCP. 

PROCEDURE 


DoD  Instruction  5000.2  delineates  the  need  and  the  format  for  both  an  SCP  and 
a  DCP.  It  is  important  that  this  justification  documentation  addresses  diagnostic  issues. 
Although  this  type  of  documentation  is  required  only  for  major  weapon  systems,  similar 
documentation  may  be  required  by  the  individual  services  at  significant  milestones.  Thus 
the  following  guidance  can  apply  to  all  weapon  systems  and  equipment.  It  is  also 
important  to  note  that  Milestone  IV,  being  a  logistic  readiness  and  support  review, 
coincides  with  the  projected  maturation  schedule  for  the  diagnostic  capability.  Thus  the 
Maturation  Program  Plan  should  relate  to  this  milestone. 
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GUIDANCE 


The  format  for  the  SOP  and  the  DCP  is  identical.  However,  the  SCP  is  primarily 
concerned  with:  (1)  program  alternative  tradeoffs;  (2)  performance/cost  and  schedule 
tradeoffs,  including  the  need  for  a  new  development  program  vs.  buying  or  adapting 
existing  U.  S.  or  Allied  military  or  commercial  systems;  (3)  appropriateness  of  the 
acquisition  strategy;  (4)  prototyping  of  the  system  or  selected  system  components;  (5) 
affordability  and  life  cycle  costs;  (6)  potential  common-use  solutions;  and  (7)  cooperative 
development  opportunities.  Thus  diagnostic  inputs  to  the  SCP  should  address  these 
factors. 


On  the  other  hand,  the  DCP  requirements  for  Milestone  II,  Full-Scale 
Development  Decision,,  should  address:  (1)  affordability,  in  terms  of  program  cost  vs.  the 
military  value  of  the  new  or  improved  system  and  its  operational  suitability  and 
effectiveness;  (2)  program  risk  vs.  benefit  of  added  military  capability;  (3)  planning  for  the 
transition  from  development  to  production,  which  will  include  independent  producibility 
assessments  (hardware/software/data  bases);  (4)  realistic  industry  surge  and  mobilization 
capacity;  (5)  factors  that  impact  program  stability;  (6)  potential  common-use  solutions;  (7) 
results  from  prototyping  and  demonstration/validation;  (8)  milestone  authorization;  (9) 
manpower,  personnel,  training,  and  safety  assessments;  (10)  procurement  strategy 
appropriate  to  program  cost  and  risk  assessments;  (11)  plans  for  integrated  logistics 
support  (DoD  Directive  5000.39);  (12)  affordability  and  life  cycle  costs;  and  (13)  associated 
command,  control,  communications,  and  intelligence  requirements,  including 
communications  security. 

The  DCP  requirements  for  Milestone  III,  Full  Rate  Production  Decision  should 
address;  (1)  results  of  completed  operational  test  and  evaluation;  (2)  threat  validation;  (3) 
production  or  construction  cost  verification;  (4)  affordability  and  life  cycle  costs;  (5)  the 
production  and  deployment  schedule;  (6)  reliability,  maintainability,  and  plans  for 
integrated  logistics  support  (DoD  Directive  5000.39);  (7)  producibility,  as  verified  by  an 
independent  assessment  (DoD  Directive  5000.38);  (8)  realistic  industry  surge  and 
mobilization  capacity;  (9)  multiyear  procurement  or  milestone  authorization;  (10) 
manpower,  personnel,  training,  and  safety  requirements;  (11)  cost  effectiveness  or  plans 
for  competition  or  dual  sourcing;  and  (12)  associated  command,  control,  communications, 
and  intelligence  requirements,  including  communications  security. 

Primary  considerations  for  Milestone  IV,  Logistic  Readiness  and  Support 
Review,  are:  (1)  logistic  readiness  and  sustainability  (peacetime  and  wartime);  (2)  weapon 
support  objectives;  (3)  the  implementation  of  integrated  logistics  support  plans,  per  DoD 
Directive  5000.39;  (4)  the  capability  of  logistics  activities  (i.  e.,  supply,  transportation,  etc), 
facilities,  training,  and  manpower  to  provide  support  efficiency  and  cost  effectively;  (5) 
disposition  of  displaced  equipment;  and  (6)  affordability  and  life  cycle  costs.  This  should 
coincide  with  the  completion  of  the  diagnostic  capability’s  maturation  period. 
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Milestone  V,  Major  Upgrade  or  System  Replacement  Decision,  is  concerned 
with:  (1)  capabiiity  of  the  system  or  facility  to  continue  to  meet  its  original  or  evolved 
mission  requirements;  (2)  the  potential  necessity  of  modifications  and  upgrades  to  ensure 
that  mission  requirements  are  met  and  that  the  useful  life  is  extended;  (3)  changes  in 
threat  that  require  increased  cap:  ability  or  utility;  (4)  changes  in  technology  that  present  the 
opportunity  for  a  significant  breakthrough  in  system  worth;  and  (5)  disposition  of  displaced 
equipment.  A  significant  question  to  be  decided  at  this  point  is  whether  deficiencies  are 
critical  enough  to  warrant  major  modification,  retirement,  and/or  new  start  considerations. 
Feedback  on  the  performance  of  the  diagnostic  capability  is  an  important  factor  in  this 
decision. 

The  above  guidance  relates  to  all  of  the  information  required  in  preparing  SCPs 
and  DCPs.  Each  of  these  items  must  be  addressed  in  relation  to  the  diagnostic  capability, 
as  appropriate.  Much  of  the  data  required  to  provide  this  information  will  be  an  output 
from  the  various  studies  conducted  by  the  contractor  as  discussed  in  this  guide  under 
Requirement  #2. 
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CHECKLIST 

Does  the  SCP/DCP  adequately  address  all  aspects  of  the 
proposed  diagnostic  capability  in  relation  to  weapon 
system  performance,  cost,  and  manpower  implications? 

□f  Does  the  SCP/DCP  contain  both  technical  and 

operational  goals/thresholds  for  diagnostic  design? 

Has  a  risk  assessment  of  the  diagnostic  capability 
been  addressed? 

E^  Do  the  results  from  the  demonstrations  relate  to  the 
demonstrated  performance  of  the  diagnostic  capability? 

E^  Does  the  DCP  relating  to  Milestone  IV  address  the 
maturation  of  the  diagnostic  capability? 


REQUIREMENTS 


REQUIREMENT  #2 


ESTABLISHING  AND  ALLOCATING  DIAGNOSTIC  REQUIREMENTS 


OVERVIEW 


Good  diagnostics  and  testabiiity  are  based  on  the  ability 
to  properly  establish  diagnostic  requirements,  which  are 
in  turn  based  on  weapon  system  mission,  the  system’s 
sustainability,  operational,  and  support  requirements 
and  the  allocation  of  these  requirements  at  system, 
subsystem,  and  unit  levels.  Lack  of  appropriate  attention 
to  this  process  results  in  diagnostic  designs  with 
questionable  basis  and  justification.  Unfortunately,  this 
process  has  not  been  transformed  from  an  art  to  a 
rigorous  methodology.  An  integrated  series  of  proven 
tools  does  not  exist  and  thus  the  quality  of  the  process 
depends  on  the  expertise  of  the  persons  performing  this 
function.  The  system  is  further  complicated  by  the 
advanced  weapon  system  architecture  which  is  now 
being  applied.  This  architecture  involves  complex 
redundancy,  reconfigurable  elements,  and 
configurations  which  allow  graceful  degradation.  A 
proper  allocation  methodology  is  an  integral  part  of 
logistic  support,  reliability,  and  maintainability  analyses 
and  is  based  on  the  weapon  system’s  mission  scenario 
and  performance  requirements.  The  analyses  are  an 
iterative  process,  which  extend  over  the  acquisition 
phases  and  often  into  deployment  of  the  weapon 
system.  Implementation  of  these  analyses  are  normally 
the  responsibility  of  the  Contractor  Program  Manager, 
with  the  results  reviewed  by  the  Government  Program 
Manager. 


TESTABILITY/ 

DIAGNOSTICS 

"n 


^  PROGRAMMATIC  | 


IMPORTANT  CONSIDERATIONS  TO  BE  ADDRESSED 

Reqmt. 

2.1  Translate  mission  and  performance  requirements  into  diagnostic 
requirements. 

2.2  Allocate  diagnostic  requirements  to  system,  subsystem  and  unit 
elements. 

2.3  Optimize  the  mix  of  diagnostic  elements. 

2.4  Assess  the  risk  of  each  diagnostic  alternative. 
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WEAPON 

SYSTEM 

ACQ.  PHASE 

OPER. 

REQMTS. 

CONCEPT 

EXPLOR. 

DEM/ 

VAL 

FSD 

PROD/ 

DEPL 

WEAPON 

SYSTEM 

ACTIVITIES 

zs  zs 

CONTRACT 

AWARD 

DIAGNOSTIC 

ACTIVITIES 

A  A 

DIAGNOSTIC  UPDATE 

REQUIREMENTS 

DIAGNOSTIC  ACTIVITY 


Diagnostic  requirements  are  identified  in  the  Concept  Expioration  Phase  from  an 
anaiysis  of  the  prime  system  mission  and  operationai  requirements. 

PROCEDURE 


The  generation  of  weapon  system  operational  requirements  is  usually 
performed  by  the  government  from  mission  studies  and  analyses  based  on  the  Statement 
of  Need  for  a  weapon  system.  The  translation  of  those  requirements  and  weapon  system 
performance  characteristics  into  diagnostic  requirements  which  are  included  in  the  system 
specification  produced  as  a  result  of  Concept  Exploration.  The  tasks  involved  in 
translating  these  requirements  may  be  performed  by  the  contractors  or  the  government 
depending  upon  the  acquisition  process  selected  during  Concept  Exploration.  For  "in- 
house"  programs,  this  task  is  performed  by  government  engineers.  Normally,  however, 
the  generation  of  diagnostic  requirements,  the  selection  and  integration  of  the  diagnostic 
elements  to  meet  these  requirements,  the  allocation  of  these  requirements  to  subsystem 
and  unit  level,  and  the  assessment  of  risk  is  performed  by  the  weapon  system 
contractors.  The  government  acts  as  a  review  agent  in  those  cases. 

The  proper  implementation  of  this  task  is  that  it  be  performed  in  conjunction  with 
the  system  engineering  and  logistic  support  analysis  process  and  include  synthesis  and 
analysis  of  the  various  mixes  of  resources  which  make  up  a  total  diagnostic  subsystem. 
The  diagnostic  requirements  analysis  process  involves  the  development  of  a  strategy  for  a 
comprehensive  diagnostic  capability  including  a  mix  of  resources  to  be  defined  for 
providing  FD/FI  capability  at  each  level  which  the  system  is  subject  to  maintenance. 
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In  order  to  translate  mission  and  operational  requirements  to  diagnostic 
capability,  it  is  important  to  postulate  a  "diagnostic  subsystem."  Characteristics  defining 
the  capability  of  the  "diagnostic  subsystem"  represent  the  results  of  the  translation.  In 
other  words,  one  must  change  mission  requirements  into  diagnostic  capability 
requirements  in  order  to  successfully  complete  this  task. 

The  diagnostic  elements  constituting  the  diagnostic  subsystem  include 
embedded  diagnostics  support  equipment  at  ail  levels  of  maintenance,  technical  data  in  all 
its  forms,  and  personnel  numbers  and  required  skill  levels. 

In  order  to  be  responsive  to  weapon  system  mission  and  performance 
requirements,  it  is  essential  that  the  translations  start  by  reviewing  all  the  requirements 
documentation  and  studies.  The  key  document  is  the  Statement  of  Need  which  contains 
the  weapon  system  mission  and  operational  requirements.  Also  important  is  the  prime 
system  architecture  concept  which  is  an  essential  element  in  the  translation  since  many 
architectural  concepts  contain  an  inherent  diagnostic  capability  that  must  be  identified  and 
addressed  early  in  the  analysis  process. 

There  are  two  key  factors  which  will  influence  the  translation  of  weapon  system 
mission  and  operational  requirements  into  diagnostic  requirements.  They  are: 

o  Specific  requirements  as  spelled  out  in  the  Statement  of  Need 

o  Available  technology. 

Analysis  of  these  specific  requirements  will  translate  requirements  for  the 
diagnostic  capability  as  well  as  constraints  on  the  diagnostic  subsystem  dictated  by  the 
operational  parameters.  The  technology  will  impact  the  inherent  diagnostic  capability  of 
the  prime  system  architecture  as  well  as  Impact  the  assessment  of  risk  of  the  final 
diagnostic  subsystem  implementation. 

Based  upon  the  above  analyses,  translation  of  mission  and  operational 
requirements  to  a  diagnostic  capability  will  result  in  a  preliminary  set  of  diagnostic 
requirements  for  the  entire  diagnostic  subsystem.  The  optimum  mix  of  diagnostic 
elements  which  constitute  the  diagnostic  capability  will  follow  the  requirements  allocation 
to  the  weapon  system,  subsystem  and  unit  levels. 

During  the  DemonstrationA/alidatlon  Phase  and  Full-Scale  Development  Phase 
the  detailed  trade  studies  will  formally  optimize  the  diagnostic  element  mix  and  provide 
implementation  specifications  for  the  diagnostic  subsystem  to  be  produced.  This  process 
is  obviously  iterative  but  most  dependent  upon  a  thorough  job  of  mission  and  performance 
requirements  analysis  and  initial  translation  into  diagnostic  requirements. 

For  example,  the  DemA/al  Phase  may  result  in  a  System  Specification  only,  with 
the  allocation  of  the  system  requirements  to  be  performed  and  redefined  in  the  FSD 
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REQUIREMENT  #2.1 


Phase.  For  some  less-than-major  systems,  DemA/al  Phase  may  be  bypassed  altogether. 
In  both  of  these  cases,  the  system-level  specifications  may  be  developed  during  FSD 
Phase.  The  analyses  described  within  this  section  should  be  performed  at  appropriate 
joints  based  upon,  and  commensurate  with,  the  level  of  detail  achieved  in  the  definition  of 
the  system  and  the  definition  of  the  support  and  maintenance  concepts  for  the  system. 

GUIDANCE 


Currently  there  is  no  formal  DoD  model  for  translating  mission  and  operational 
requirements  into  a  diagnostic  capability.  However,  using  system  engineering  approaches 
defined  in  MIL-STD-499,  the  contractor  can,  indeed,  deveiop  an  initial  set  of  diagnostic 
subsystem  requirements  which  are  traceable  to  weapon  system  requirements,  weapon 
system  priorities,  and  avaiiable  technology. 

Success  in  translating  mission  and  operational  requirements  into  diagnostic 
requirements  is  embodied  in  the  ability  to  develop  higher  order  measures  for  defining 
weapon  system  characteristics  that  relate  to  fault  detection  and  fault  isolation  parameters. 

Typical  weapon  system  characteristics  which  must  be  evaluated  include  the  following; 


Probability  of  Mission  Success 

Availability 

Utilization  Rate 

Population 

Turnaround  Time 

Threat 

Mobility 

Safety 

Alert 


Deployment 

Basing 

Weight 

Repair  Concept 

Personnel 

Training 

Cost 

Etc. 


The  following  weapon  system  priorities  are  of  major  concern: 

War  fighting  capability 

Survivability 

Mobility 

Manpower 

Life  Cycle  Cost. 


During  the  Concept  Exploration  Phase,  mission-oriented  measures  are 
overriding  for  diagnostic  requirements  generation.  Resource  criteria  (manpower,  cost, 
facilities,  etc.)  become  significant  during  synthesis  of  specific  diagnostic  element  mixes. 

The  mission  data  to  be  collected  and  considered  for  generating  the  diagnostic 
requirements  is  as  follows: 
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REQUIREMENT 


o  Mission  scenarios  definition  (prioritized  in  order  of  criticality) 

o  Mission  rate/length 

o  Mission  operation  (continuous  vs.  intermittent) 

o  Mission  phases 

o  Time  demands  and  operational  constraints  per  mission  phase 

o  Subsystem/function  utilization  per  mission  phase  (survivability  or  safety 
critical) 

o  Functions/failures  impacting  personnel  safety 

o  Functions/failures  impacting  system/equipment  safety  (sustainability  or 
mission  critical) 

o  Functions/failures  impacting  mission  success  (per  mission  phase). 

A  key  diagnostic  parameter  to  be  determined  through  the  analysis  of  mission 
requirements  is  the  maximum  failure  latency  per  operating  function  for  each  mission 
phase.  This  parameter  will  drive  the  fault  detection  requirements  which,  in  turn,  serve  as 
the  basis  for  BIT  design.  Failure  latency  Is  the  elapsed  time  between  fault  occurrence  and 
failure  indication.  Maximum  failure  latency  is  the  maximum  allowable  time  between  the 
occurrence  of  a  fault  and  the  reporting  or  "handling"  of  the  failure.  As  a  simplistic  example, 
if  a  fire  control  system  fault  occurs,  and  the  fire  control  system  function  is  highly  critical  to 
mission  success,  then  the  maximum  failure  latency  will  be  very  small  --  perhaps  expressed 
in  microseconds  or  nanoseconds.  The  fault  detection  (FD)  time  requirement  will  reflect  the 
failure  latency  factor  --  thereby  driving  the  BIT  technique  to  provide  concurrent 
performance  monitoring.  Fault  tolerance  through  redundancy  may  be  required  or 
considered.  This  simplistic  example  is  made  more  complex  by  factoring  in  the  time 
demands  per  mission  phase  of  the  fire  control  system.  It  is  made  still  more  complex  by 
factoring  in  operating  anomalies  and  intermittents  into  the  FD  requirements. 

In  the  definition  of  diagnostic  requirements,  it  is  important  to  note  that  the 
diagnostic  capability  is  made  up  of  the  inherent  diagnostic  capability  of  the  prime  system, 
as  well  as  added  diagnostic  elements.  It  is  therefore  important  that  diagnostic  analysis  be 
integral  to  the  prime  weapon  system  engineering  process,  since  performance  and  support 
parameters  can  no  longer  be  isolated  from  design. 

The  prime  configuration  represents  a  performance  capability.  The  niission 
requirements  can  be  related  directly  to  the  configuration  by  analysis  of  the  behavior  of  the 
utilized  configuration  items  over  the  time  demands  imposed  by  mission.  A  representation 
of  performance  over  time  can  be  easily  presented  to  management  for  setting 
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J 


requirements.  This  measure  is  referred  to  as  P  (Performance  time  dependency),  which  is 
described  in  MIL-HDBK-338.  P  can  be  calculated  and  plotted  using  equations  for  mission 
reliability  in  MIL-STD-756. 

Operational  constraints  also  must  be  addressed.  The  checklist  below  presents 
the  operational  data  to  be  collected  and  considered  in  diagnostic  requirements  analysis. 

o  Environmental  conditions  (temperature,  rain,  dirt,  salt  spray,  etc.).  Applies  to 
both  the  prime  system  and  support  equipment. 

o  Operating  locations  (dispersed  vs.  centralized) 
(remote/accessible/inaccessible) 

o  Space  limitations  (for  personnnel  and/or  test  equipment) 

o  Mobile  or  fixed  maintenanace  facilities 

o  Independent  operation  or  part  of  a  battle  group 

o  Manpower  constraints  (number  and  skill  levels). 

The  constraints  under  which  a  weapon  system  will  operate  must  be  identified 
and  evaluated  in  terms  of  the  impact  on  testability  requirements.  System  design  and 
supportability  factors  must  take  into  account  these  constraints.  Operating  constraints  will 
often  drive  the  diagnostic  strategy  to  use  of  embedded  versus  external  test  resources. 

Prime  System  Architecture/Configuration 

Data  must  be  collected  on  the  architecture  and  configuration  alternatives  of  the 
prime  system  to  be  developed  with  respect  to  partitioning.  Interconnections  and  flow  as 
input  to  the  testability  requirements  analysis.  The  architectures  under  consideration  will 
have  inherent  characteristics  which  may  support  or  impede  diagnostics.  The 
performance  capability  of  alternative  prime  system  architectures  must  be  evaluated 
against  the  mission  requirements,  time  phases  and  equipment  utilization/demands. 

It  is  useful  for  this  evaluation  to  plot  curves  of  capability  vs.  time  demands 
imposed  by  the  mission.  The  resulting  P  (Performance  over  Time)  curve  can  include 
resource  constraints  (spares,  personnel)  and  operational  constraints  (maximum  allowable 
repair  time). 

The  following  prime  system  configuration  data  should  be  collected  for  input  to 

this  step: 

o  Work  Breakdown  Structure  (MIL-STD-881) 
o  List  of  government  furnished  equipment/ 
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off-the-shelf  equipment/  non-developmental  items 
(for  above,  item  or  candidate  Hem) 
o  Prime  system  archHecture  alternatives 
0  InHial  failure  rate  projections  and  characterization 
o  Fault-tolerant  or  redundant  functions 
o  Technologies  to  be  used  (H  known) 
o  Level  of  integration  vs.  autonomy. 

Based  upon  analysis  of  architectures  under  consideration,  high-level  diagnostic 
opportunities  should  be  identified.  This  includes  incorporation  of  a  test  and  maintenance 
bus,  fault-tolerant  design  coordination,  system-level  diagnostic  resources  -  such  as  data 
acquisition/collection  subsystems  or  embedded  adaptive  diagnostic  subsystem  and  use  of 
standard  diagnostic  connections  and  interfaces. 

Diagnostic  inputs  must  be  made  within  the  system  engineering  process  prior  to 
the  final  selection  of  the  prime  system  archHecture. 

Evaluate  Technology  Opportunities 

Advanced  diagnostic  technology  opportunity  or  implications  must  be  identified 
based  on  the  following  areas: 

o  Baseline  comparison  system  major  drivers,  supportabiiity  problem  areas, 
targets  of  improvement 

o  Incorporation  of  LSI,  VLSI,  VHSIC,  expert  system,  or  other  advanced  design 
technology  in  system 

o  Need  to  improve  requisite  operational  capability  having  no  prior  design 
solution. 

Examples  of  advanced  diagnostic  technology  opportunities  which  may  be 
exploitable  on  the  new  system  include: 

o  Expert  system  based  maintenance  aids 

o  Test  and  Maintenance  bus  concepts 

o  "Smart"  BIT  techniques 

o  Adaptive  diagnostic  subsystems 

o  Prognostics  concepts 

o  Automated  technical  information  authoring 
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o  Advanced  packaging  techniques 

o  Advanced  instrumentation  (stimulus  and  measurement)  technologies 

0  Automatic  capture  of  CAD  data  for  diagnostics  generation. 

Upon  determination  of  advanced  technology  applications,  inputs  must  be  made 
to  the  design  engineering  effort  regarding  design  constraints  related  to  the  above 
concepts. 

Diagnoetic  Element  Constraints 

In  order  to  define  specific  diagnostic  characteristics  and  requirements  of 
the  system  or  to  further  "close  in"  the  envelope  within  which  tradeoffs  are  conducted, 
diagnostic-related  constraints  are  established.  This  includes  constraints  placed  on 
built-in  test  design  attributes  and  functions,  testability  constraints  and  test  equipment 
constraints.  This  may  also  Include  broader  diagnostic-related  constraints,  such  as 
page  count  of  technical  information  or  maintenance  technician  skill-levels.  These 
constraints  are  driven  by  mission  requirements,  design,  operation  and  support 
characteristics,  or  standardization  policies  imposed. 


Sample  diagnostic-related  constraints  are  provided  below. 


Driving  System  Requirement 

Resulting  Diagnostic  Constraint/Requirement 

Mission  Requirement 

Mobility 

Test  Equipment  Size/Weight 

Continuous  Operation 

BIT  Interface  Planned  Maintenance  Duty  Cycle 

Sustainabiiity 

Redundancy 

Reconfigurability 

Fault-Tolerant  Design 

Standardization  Imposed 

Standard  Test  Equipment 

Standard  Diagnostic  Connectors,  Controllability/Observability 
Interface  to  UUT 

QFE 

BIT  design/capabilities 

Standard  Bus 

Interface  Design/Profocol 

Desiqn  Characteristics 

Power  Availability 

BIT  Power  Consumption 

System  Weight 

BIT  and  Test  Connector  Weight 

System  Size 

Volume  of  BIT  Circuitry  and  Test  Connectors 
(Real  estate  available  for  BIT  circuitry) 

Volume  required  for  increased  modularity 

Memory  Limitation 

Memory  aifocatable  to  BIT  functions 

Operating  System  Char. 

Software  BIT  function  constraints 

Cost 

Cost  of  additional  hardware  required  for  BIT  and  testability 
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Establish  Diagnostic  Objectives 

Analysis  of  weapon  system  data  ascertained  must  be  performed  to  identify 
diagnostic  objectives  based  on  system  requirements.  Diagnostic  objectives  to  be 
considered  include: 

o  BIT  FD/R  requirements  to  support  preliminary  maintenance  concept 

-  Repair  Times 

-  Reconfigurability 

-  Deferred  Maintenance 

-  Fault  Tolerance 

o  BIT  requirements  to  support  system  confidence  checks 
o  Requirement  to  deal  with  intermittent  faults  or  operational  anomalies 
0  Prime  system  architecture  testability  opportunities 
o  GFE  testability  factors/constraints 
o  Requirements  for  vertical  testability. 

Examples  of  typical  objectives  to  be  established  at  this  point  are  provided  below. 
DRIVING  SYSTEM  FACTOR  SAMPLE  DIAGNOSTIC  OBJECTIVE 


Maximum  Acceptable  Failure  Latency — =» 

Mission/Safety  Critical  Function - => 

MTTR,  Spares - => 

Manpower  and  Skill  Levels - => 

QFE  Constraints - =» 

Fault-Tolerant  Design  Coordination — => 

2  Level  Maintenance - => 

Life  Cycle  Cost  Priorities - => 

Minimize  RTOK  Rate  Between - 

Maintenance  Levels 


Fault  Detection  Time 
Performance  Monitoring 
Fault-Isolation  Level 
BIT  Fault- Isolation  Level 
System-Level  BIT  Requirements 
Performance  Monitoring 
Ambiguity  Group  Size/ATE  Size  &  Weight 
Reliance  on  Embedded  Diagnostics 
Utilize  Compatible  Test  Equipment, 
Techniques,  Tolerances 


Initial  diagnostic  requirements  result  from  analysis  of  weapon  system 
characteristics,  prioritized  as  needed,  on  diagnostic  elements.  It  is  convenient  to  partition 
the  diagnostic  elements  as  embedded  and  external. 


Some  of  the  tradeoffs  to  be  made  for  generating  embedded  and  external 
diagnostic  requirements  include: 


o  Functions  and  level  of  built-in  test  vs.  external  diagnostics 
o  Functional  vs.  parametric  testing 

o  Built-in  test  fault  detection 

-  Concurrent  performance  monitoring 

-  Periodic  BIT  routines 
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-  Operator  initiated  BIT  routines 

o  Level  of  diagnostic  capability  at  each  level  of  maintenance  (e.g.,  detect  95% 
of  faults;  isolate  to  3  LRUs;  within  30  minutes) 

o  Diagnostic  elements  to  be  used  at  each  level  of  maintenance  (e.g.,  test 
equipment,  technicai  information  and  maintenance  aids,  training  and  skiii 
.  ieveis). 

Once  the  ievei  of  built-in  test  is  estabiished,  a  maintenance  workload  generated 
by  operational  and  failure  rate  data  can  be  projected.  At  this  point,  detailed  tradeoffs  can 
be  performed  regarding  the  optimization  of  testability,  including  level  of  diagnostic 
capability  at  each  level  of  maintenance,  and  the  effectiveness  and  efficiency  of  the  mix  of 
diagnostic  elements  to  be  used  at  each  level  of  maintenance.  A  baseline  comparison 
system  is  used  to  project  failure  data.  The  requirements  that  need  to  be  established  are 
outlined  below: 

Embedded  Diagnostic  Requirements 

0  System  Integrated  Test  (SIT)  requirement  for  monitoring  of  mission-critical 
functions  and  functions  affecting  personnel  safety  (derived  from  maximum 
allowable  failure  latency) 

o  BIT/SIT  requirements  to  support  operational  constraints 

o  Requirement  to  deal  with/handle  Intermittents/anomalies 

o  BIT/SIT  requirements  to  support  system  confidence  checks 

o  Prime  system  architecture,  testability  opportunities,  and  GFE  testability 
factors/constraints 

o  BIT  requirements  to  support  preliminary  maintenance  concept,  based  on: 

-  Level-of-repair  analysis 

-  Manpower  available 

-  Skill  levels  available/required 

-  Deferred  maintenance  goals 

-  Repair  times  (driving  fault  isolation  time) 

-  Sparing  concepts  (driving  fault  isolation  levels) 

-  Standardization  requirements/goals  (test  equipment,  personnel 
qualifications) 

o  Requirement  to  provide  handshake  to  external  diagnostic  resources  (vertical 
testability,  vertical  diagnostics). 
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External  Diagnostic  Requirements  (Support  Equipment,  Technical  Data,  and 
Personnel) 

OPERATIONALVORGANIZATIONAL  MAINTENANCE  LEVEL: 

o  Requirement  for  elements  to  optimize  interface/utilization  of  embedded 
diagnostic  elements 

o  Define  FD/FI  functions  to  satisfy  O-Level  maintenance  operations  (driven  by 
inputs  from  operational  constraints  and  preliminary  maintenance  concept), 
based  on: 

-  Minimization  of  unnecessary  removals 

-  Mobility  requirements/space  available 

-  Level-of-repair  analysis 

-  Sustainability 

-  Manpower  available 

-  Skill  levels  available/required 

-  Repair  times 

-  Sparing  concepts 

-  Standardization  requirements/goals 

o  O-Level  technical  information  (including  maintenance  aids) 
o  O-Level  test  equipment 

-  Manual  test  equipment 

-  Automatic  test  equipment  and  test  programs 

-  Portable  maintenance  aids 

o  O-Level  training  requirements  to  support  skills  required 

-  On-the-job  training 

-  Formal  school  training 

o  O-Level  data  acquisition/collection  system  (and  data  management) 

o  Requirements  to  provide  O-Levei  handshake  to  1-Level  diagnostic  elements 
(vertical  testability,  vertical  diagnostics) 
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INTERMEDIATE  MAINTENANCE  LEVEL: 

o  Define  FD/FI  functions  to  satisfy  l-Level  maintenance  operations  based  on: 

-  Minimization  of  unnecessary  removals 

-  Mobility  requirements/space  available 

-  Level-of-repair  analysis 

-  Sustainability  of  spares  pipeline 

-  Manpower  available 

-  Skill  levels  available/required 

-  Repair  times 

-  Sparing  concepts 

-  Standardization  requirements/goals 

o  l-Level  technical  information  requirements  (including  maintenance  aids) 
o  l-Level  test  equipment  requirements 

-  Manual  test  equipment 

-  Automatic  test  equipment  and  test  program  sets 

o  l-Level  training  requirements  to  support  skills  required 

-  On-the-job  training 

-  Formal  school  training 

o  l-Level  data  acquisition,  collection,  management,  analysis,  processing 
system  requirements 

o  Requirement  to  provide  l-Level  handshake  to  Depot-Level  diagnostic 
elements  (vertical  testability) 

DEPOT  MAINTENANCE  LEVEL: 

o  Define  FD/FI  functions  to  satisfy  Depot-Level  maintenance  operations, 
based  on: 

-  Level-of-repair  analysis 

-  Sustainability  of  spares  pipeline 

-  Manpower  availability 

-  Skill  levels  available/required 

-  Repair  times 

-  Sparing  concepts 

-  Standardization  requirements/goals 
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o  D-Level  technical  information  requirements  (inciuding  maintenance  aids) 

o  D-Levei  test  equipment  requirements 

-  Manuai  test  equipment 

-  Automatic  test  equipment  and  test  program  sets 

o  D-Level  training  requirements  to  support  skills  required 

-  On-the-job  training 

-  Formai  schooi  training 

0  D-Level  maintenance  data  acquisition,  coilection,  analysis,  processing 

o  Requirement  to  capture  and  utilize  factory  test  resources  and  results  and/or 

data  for  Depot  use  (vertical  testability,  vertical  diagnostics). 

Since  the  overall  diagnostic  capability  must  be  defined,  quantified,  designed, 
evaluated,  etc.,  it  is  best  defined  as  a  "diagnostic  subsystem."  This  subsystem  can  be 
broken  down  into  its  component  parts  and  defined  in  a  type  of  format.  This  format  will 
facilitate  the  hierarchical  allocation  and  diagnostic  mix  optimization  process  because 
function  and  cost  parameters  can  be  quantitatively  assigned  to  each  element.  Alternative 
diagnostic  subsystems  may  then  be  easily  synthesized  and  evaluated. 
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CHECKLIST 

O'  Have  the  relative  weapon  system  priorities  been 

overlayed  on  the  specific  weapon  system  characteristics 
established  in  the  Statement  of  Need? 

O'  Has  the  inherent  diagnostic  capability  of  the  prime 
system  architecture  been  included  in  the  analysis? 

O'  Have  the  requirements  been  generated  for  both 
embedded  and  external  diagnostics? 

O'  Has  the  mission  and  operational  data  been  utilized 
in  the  diagnostic  requirements  generation? 
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WEAPON 
SYSTEM 
ACQ.  PHASE 


WEAPON 

SYSTEM 

ACTIVITIES 


OPER. 

REQMTS. 


CONCEPT 

EXPLOR. 


DEM/ 

VAL 


PROD/ 

DEPL 


CONTRACT 

AWARD 


DIAGNOSTIC 

ACTIVITIES 


ALLOCATION  UPDATE 


DIAGNOSTIC  ACTIVITY 

Allocalion  of  diagnostic  requirements  to  the  system,  subsystem,  and  unit  level  is 
required  in  order  to  assign  specific  specification  values  to  each  configuration  item  which 
forms  part  of  the  weapon  system.  The  allocation  process,  normaily  performed  by  the 
contractor,  shall  assure  that  the  weapon  system  diagnostic  requirements  and  the 
constraints  on  the  diagnostic  subsystem  are  not  violated  during  the  "flow  down"  process. 

PROCEDURE 

Initial  allocation  of  diagnostic  requirements  to  lower  system  levels  must  be 
based  upon  the  time  demands  placed  upon  the  system  configuration  by  the  mission 
requirements. 

After  the  initial  set  of  diagnostic  requirements  has  been  defined,  a  diagnostic 
mix  is  postulated  from  the  synthesized  diagnostic  subsystem  alternatives  in  order  to 
implement  the  initial  set  of  diagnostic  requirements. 

Whereas  the  initial  diagnostic  requirements  are  driven  by  mission  time 
demands,  the  optimization  of  the  diagnostic  element  mix  is  driven  by  resource  constraints. 
Simply  stated,  the  requirements  generation  process  indicates  what  is  needed  and  the 
diagnostic  mix  generation  process  indicates  the  affordable  solutions.  A  risk  analysis 
performed  during  the  subsequent  phases  of  system  development  confirm  the  solutions  as 
feasibie.  It  is,  therefore,  important  to  note  that  the  allocation  procedure  is  a  partial  step  in 
the  development  of  a  diagnostic  system.  During  the  diagnostic  element  optimization  and 
design  process,  it  may  be  cost  effective  to  reallocate  the  diagnostic  requirements  in  order 
to  achieve  better  implementations  with  respect  to  resource  constraints.  Many  of  these 
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tradeoffs  are  driven  by  both  technology  and  the  acquisition  business  decisions  that  are 
made  for  each  weapon  system  program.  For  example,  allocation  of  a  testability  strategy 
to  each  subsystem  may  not  be  feasible  due  to  the  existence  of  many  government- 
furnished  equipments  within  a  particular  weapon  system.  In  those  cases,  a  centralized 
system-level  test  approach  may  be  more  desirable.  A  shift  in  allocation  from  subsystem  to 
system  level  will  prove  effective  in  the  implementation  of  that  particular  weapon  system 
diagnostics. 

To  achieve  this  flexibility,  the  allocation  process  must  be  tied  to  the  system-level 
reliability  and  maintainability  models.  This  model  will  contain  the  allocated  parameters 
with  traceability  back  to  system-level  parameters.  In  this  way,  as  the  program  proceeds 
from  Concept  through  Dem/Val  into  Full-  Scale  Development,  each  of  the  values  can  be 
traded  off  as  the  diagnostic  subsystem  is  configured  and  optimized  as  a  result  of 
knowledge  gained  from  trade  studies. 

GUIDANCE 


A  preliminary  diagnostic  allocation  should  be  prepared.  The  allocation  should 
include  all  diagnostic  elements  and  consideration  of  all  maintenance  levels.  The  allocation 
of  diagnostic  goals/values  should  be  accomplished  through  the  application  of  structured 
processes,  based  on  task  description  and  guidance  provided  within  applicable  military 
standards.  The  tasks  and  guidance  paragraphs  that  define  the  allocation  process  to  be 
employed  are: 


MIL-STD-499 

Task  10.2.3 

Allocation 

MIL-STD-785 

Task  202 

Reliability  Allocation 

MIL-STD-470 

Task  202 

Maintainability  Allocation 

MIL-STD-2165 

Task  201 

Testability  Requirements. 

The  two  fundamental  tools  used  during  the  allocation  process  are  the  weapon 
system  detailed  life  cycle  cost  model  and  the  weapon  system  reliability  functional  flow 
diagrams.  During  early  Concept  Exploration  these  tools  are  based  upon  baseline 
comparison  systems  and,  subsequently,  evolve  into  the  actual  proposed  configuration. 
The  allocation  of  diagnostic  requirements  is  usually  performed  as  part  of  the  overall  LSA 
process  with  strong  integration  with  reliability  and  maintainability  disciplines. 

It  is  important  that  this  allocation  process  include: 

o  FD/FI  coverage  for  all  (100%)  faults  known  or  expected  to  occur  at  each 
maintenance  level,  and 

o  Quantification  of  all  diagnostic  elements. 
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Figure  4  is  a  Notional  Diagnostic  Allocation  Specification,  which  exemplifies 
these  concepts.  This  allocation  process  is  also  closely  tied  to  the  optimization  process 
(Requirement  #2.3).  it  is  important  that  this  allocation  process  include  quantification  of  all 
diagnostic  elements.  For  instance,  the  time  to  access  technical  information  can  determine 
whether  paper  or  electronic  delivery  of  technical  information  is  required.  Formal  training 
time  may  influence  the  need  for  on-the-job  training  aids. 

This  system-level  allocation  forms  the  basis  for  the  System  Specification 
discussed  under  Requirement  #1 .2.  It  also  Is  followed  by  allocation  down  to  subsystem 
and  item  levels. 
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FIGURE  4.  Notional  Diagnostic  Allocation  Specification 
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I 


Allocate  Requirements  to  Item  Development  Specification 

System-level  diagnostic  requirements  are  allocated  down  to  subsystem  and  item 
levels  for  the  purpose  of  the  development  of  those  items.  Diagnostic  requirements  for 
Configuration  Items  (Cl)  support  two  distinct  requirements:  system  test  (primarily  BIT)  and 
shop  test  (ATE  and  GPETE). 

Quantitative  testability  requirements  for  each  Configuration  Item  are  allocated 
from  system  diagnostic  requirements  based  upon  FMEA  data,  relative  failure  rates  of  CIs, 
mission  criticality  of  the  CIs,  what  is  achievable  for  each  Cl  or  other  specified  criteria.  The 
failure  detection  level  of  the  Cl  is  weighted  by  the  items’  failure  rate  to  ensure  that  system- 
level  fault  detection  capability  is  achieved;  Table  2  below  is  an  example  of  an  allocation  of 
a  system-level  BIT  fault  detection  requirement  which  is  allocated  to  five  configuration 
items.  The  table  shows  three  alternative  FD  allocations  which  meet  the  system-level  BIT 
FD  requirement  of  95%. 


TABLE  2.  SAMPLE  ALLOCATION  OF  95%  BIT  FD  REQUIREMENT 


CONFIQURATION 

ITEM 

IIIQIfl 

FD  ALLOCATION 
#1 

FD  ALLOCATION 
#2 

FD  ALLOCATION 
#3 

A 

100 

.95 

.98 

.95 

B 

10 

.95 

.80 

C 

50 

.95 

.70 

.98 

D 

200 

.95 

.99 

.90 

E 

100 

.95 

.98 

.99 

SYSTEM 

460 

.95 

.95 

.95 

The  BIT  performance  capabifity  and  testability  characteristics  of  GFE  portions  of 
the  system  should  be  considered  in  the  allocation.  For  example,  assume  a  GFE  item  with 
only  70%  BIT  fault-detection  capability.  In  order  to  accomplish  the  95%  system-level 
capability  required  in  the  above  example,  the  allocation  distribution  must  take  into  account 
the  capability  of  each  of  the  items  which  make  up,  or  contribute  to,  the  system  level.  The 
capability  of  the  GFE  then  serves  as  a  constraint  in  the  allocation.  In  the  above  example, 
given  that  Item  C  is  GFE  with  70%  BIT  fault-detection  capability,  the  FD  allocation  scheme 
#2  is  a  real  world  alternative  and  the  others,  #1  and  #3,  are  not. 


Shop  test  requirements  are  determined  by  how  the  Cl  is  further  partitioned,  if  at 
all,  into  Units  Under  Test  (UUT).  Diagnostic  requirements  for  each  UUT  should  be 
included  in  the  appropriate  Cl  Development  Specification.  These  parameters  are  not 
allocated  from  the  system-level  requirements,  but  rather  are  driven  by  the  diagnostic 
concept  of  off-line  test  requirements  of  the  Configuration  Items. 

In  many  digital  systems,  built-in  test  is  implemented  in  whole  or  in  part  through 
software.  Here,  diagnostic  requirements  will  appear  in  a  Computer  Program  Configuration 
Item  (CPCI)  development  specification.  The  CPCI  may  be  dedicated  to  the  built-in  test 
function  (i.e.,  a  maintenance  program)  or  may  be  a  mission  program  which  contains  test 
functions. 
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CHECKLIST 

Were  the  system  reliability  and  maintainability  models 
used  in  the  diagnostic  allocation  process? 

Are  the  allocated  values  traceable  to 
Weapon  System  Requirements? 

Were  constraints  allocated  to  all  diagnostic  elements? 
O'  Were  constraints  assigned  to  all  maintenance  echelons? 


OPTIMIZATION  OF  THE  DIAGNOSTIC  ELEMENT  MIX 


REQUIREMENT  #2.3 


WEAPON 

SYSTEM 

ACQ.  PHASE 

OPER. 

REQMTS. 

CONCEPT 

EXPLOR. 

DEM/ 

VAL 

FSD 

PROD/ 

DEPL. 

WEAPON 

SYSTEM 

ACTIVITIES 

zs  zs  zs 

^ - - - PRELIMINARY 

CONTRACT  DESIGN 

AWARD 

DIAGNOSTIC 

ACTIVITIES 

A  A 

OPTIMIZE  UPDATE  UPDATE 

DIAG.  MIX  OPTIMIZE  MIX 

DIAGNOSTIC  ACTIVITY 

Given  the  allocation  of  diagnostic  requirements  to  the  subsystem  and  unit  level, 
the  "diagnostic  subsystem"  must  be  defined  as  part  of  the  overall  weapon  system 
specification.  The  resulting  diagnostic  subsystem  includes  both  embedded  and  external 
support.  External  support  must  be  defined  at  all  levels  of  maintenance  and  includes 
technical  information,  support  equipment,  and  personnel  numbers  and  skill  levels. 
Verification  of  the  result  of  this  process  is  the  job  of  the  Government  Program  Manager. 

PROCEDURE 

The  starting  point  for  developing  the  diagnostic  subsystem  is  the  generation  of  a 
diagnostic  subsystem  profile  from  the  weapon  system  characteristics  and  priorities.  Each 
of  the  diagnostic  elements  will  have  a  differing  impact  on  the  weapon  system 
characteristics.  For  example,  a  high  priority  constraint  on  logistic  support  would  favor  a 
high  degree  of  embedded  diagnostics.  On  the  other  hand,  constraints  on  personnel  may 
favor  technical  information  systems  with  a  high  degree  of  artificial  intelligence.  Analysis  of 
the  weapon  system  characteristics  in  terms  of  their  impact  on  the  support  elements  will 
generate  various  support  element  diagnostic  profiles. 

The  diagnostic  profiles  will  portray  various  mixes  of  diagnostic  elements  for 
different  weapon  system  characteristics  and  constraints.  Each  of  the  diagnostic  element 
profiles  infers  a  diagnostic  subsystem  which  can  be  built  and  delivered  with  the  weapon 
system.  The  optimization  issue  is  the  selection  of  a  diagnostic  subsystem  which  can  be 
implemented  at  low  risk  and  which  meets  the  requirements  allocated  to  system, 
subsystem,  and  unit  level. 

i 
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The  key  to  optimization,  therefore,  is  the  development  or  synthesis  of  various 
alternative  diagnostic  subsystems  based  upon  the  weighted  diagnostic  element  profiles. 
This  is  an  engineering  task  and  represents  an  important  aspect  in  the  overall  development 
of  a  diagnostic  capability  for  the  weapon  system.  By  generation  of  a  diagnostic 
subsystem,  early  in  Concept  Exploration,  the  overall  design  integration  of  support  and 
prime  design  elements  will  be  achieved.  During  the  Dem/Val  and  Full-Scale  Development 
Phases,  the  diagnostic  subsystem  is  refined  based  upon  trade  studies. 


The  key  is  to  identify  the  sensitivity  of  the  various  diagnostic  element  function 
contributions  to  the  overall  life  cycle  costs,  and  to  ensure  that  all  diagnostic  functional 
requirements  are  considered  and  included  as  part  of  the  total  diagnostic  subsystem 
synthesis. 


Each  diagnostic  subsystem  alternative  synthesized  is  evaluated  with  respect  to; 

o  Impact  on  Mission  Performance  Over  Time 

o  Impact  on  Resource  Requirements 

-  Acquisition  Cost 

-  Life  Cycle  Cost 

-  Manpower  Requirements 

o  Responsiveness  to  operational  constraints. 

The  evaluation  is  performed  by  assigning  values  related  to  the  evaluation 
factors  listed  above  to  the  diagnostic  subsystem  or  to  the  elements  of  the  diagnostic 
subsystem. 

To  evaluate  the  diagnostic  subsystem  impact  on  mission  performance  over  time, 
the  diagnostic  capability  values  relate  to  diagnostic  performance  provided  by  the  elements 
per  mission  phase,  duration  and  equipment  utilization. 

To  evaluate  the  responsiveness  of  the  diagnostic  subsystem  to  operational 
constraints,  the  operational  constraints  must  be  assigned  qualitative  or  quantitative  values. 
The  impact  of  the  diagnostic  subsystem  characteristics  on  those  values  (time  demands) 
must  then  be  determined.  This  analysis  includes  availability  parameters  as  well  as 
mission  reliability  calculations  based  upon  the  stated  time  demands  and  subsystem 
utilization.  The  system  reliability  model  is  a  very  effective  and  available  tool  for  this 
analysis. 


To  evaluate  the  impact  of  the  diagnostic  subsystem  on  resources,  cost  factors 
must  be  assigned  to  each  element  of  the  diagnostic  subsystem.  Each  element  of  the 
diagnostic  subsystem  can  be  assigned  a  cost  factor.  Non-recurring  (development)  and 
recurring  (production  and  support)  costs  must  be  considered.  T'  e  manpower 
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requirements  associated  with  the  alternative  diagnostic  subsystems  must  be  evaluated. 
Existing  LCC  models  should  be  used  in  this  analysis.  Data  items  should  be  standardized 
wherever  possible. 

The  cost  deltas  associated  with  each  alternative  must  be  evaluated  with  respect 
to  the  off-line  maintenance  workload  costs  and  efficiencies  generated  by  the  alternative 
embedded  diagnostic  subsystems.  A  key  diagnostic  element  workload  driver  is  ambiguity 
group  size  and  RTOK  rates. 

Based  upon  the  evaluations  performed,  the  optimum  diagnostic  subsystem 
alternative  is  selected  and  the  weapon  system  diagnostic  concept  is  estabiished  and 
documented.  The  diagnostic  concept  includes  prime  system  architecture  considerations, 
BIT  requirements  at  the  systeni  and  subsystem  levels,  test  equipment,  technical 
information  and  personnel  and  training  requirements  for  each  level  of  maintenance.  The 
diagnostic  function  of  each  element  must  be  clearly  and  quantitatively  defined  as  a 
diagnostic  requirement. 

Utilizing  the  above  procedure,  the  result  of  the  optimization  process  is  the 
development  of  a  diagnostic  subsystem  early  in  Concept  Exploration.  This  parailels  the 
development  effort  for  radar  subsystems,  fire  control  subsystems,  etc.  The  diagnostic 
subsystem  becomes  a  weapon  system  attribute  early  in  Concept  Exploration  and 
continues  to  evolve  during  subsequent  program  phases. 

GUIDANCE 


There  is  no  formal  guidance  available  for  synthesis  and  optimization  of  a 
diagnostic  subsystem.  A  generic  hierarchical  view  of  a  diagnostic  subsystem  which 
includes  engineering  and  program  management  disciplines  as  well  as  embedded  and 
external  support  elements  is  included  below  to  serve  as  guidance  for  the  contractors.  This 
indentured  diagnostic  subsystem  breakdown  will  allow  costing  by  the  contractor  for  various 
alternatives  proposed  to  satisfy  the  diagnostic  requirements  which  have  been  allocated  at 
all  system  levels.  As  experience  data  is  accumulated  on  diagnostic  subsystem 
effectiveness  and  cost,  it  will  be  possible  to  predict  many  of  these  values  early  in  Concept 
Exploration  using  the  diagnostic  profile. 

DIAGNOSTIC  SUBSYSTEM  HIERARCHY 
I.  PROGRAM  MANAGEMENT/ENGINEERING 

.A.  Requirements  Analysis 

B.  Diagnostic  Design  &  Analysis/Assessment 

C.  System  Integration  &  Test 

D.  Maturation  Program 
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II.  EMBEDDED  DIAGNOSTIC  ELEMENTS 

A.  System-Level  Diagnostic  Elements 

1 .  System-Level  Diagnostic  Hardware 

a.  Test  and  Maintenance  Bus 

b.  Sensors/Monitors 

c.  Diagnostic  Panel/Display 

d.  Embedded  ATE 

2.  System-Level  Diagnostic  Software 

a.  Status  Monitoring 

b.  Self  Test/Expert  Systems 

c.  Prognostics 

d.  Reconfigurability 

3.  Diagnostic  Data  Collection/Analysis  System 

B.  Subsystem  Diagnostics 

1 .  Subsystem  "A"  BIT 

a.  BIT  Hardware 

1 .  On  Chip 

2.  On  Printed  Circuit  Board 

b.  BIT  Software  &  Firmware 

c.  Interface  to  T&M  Bus 

2.  Subsystem  "B"  BIT  (Radar),  etc. 


III.  EXTERNAL  DIAGNOSTIC  ELEMENTS 

A.  O-Level  Diagnostics 

1 .  Technical  Information 

a.  Maintenance  Aids 

b.  Paper-Based  Manuals 

c.  Diagnostic  Data  Base 

2.  Test  Equipment 

a.  Manual  Test  Equipment 

b.  Automatic  Test  Equipment 

1 .  ATE  Hardware 

2.  Diagnostic  Software 

a.  Expert  Systems 

b.  Test  Program  Sets  (TPS) 
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3.  ATE/ILS 

3.  Trained  Personnel 

a.  Manpower 

b.  Skills 

1.  Formal  Training 

2.  On-The-Job  Training 

4.  Diagnostic  Data  Collection/Analysis  System 

B.  1-Level  Diagnostics 

1.  Technical  Information 

a.  Maintenance  Aids 

b.  Paper-Based  Manuals 

c.  Diagnostic  Data  Base 

2.  Test  Equipment 

a.  Manual  Test  Equipment 

b.  Automatic  Tost  Equipment 

1 .  ATE  Hardware 

2.  TPS 

3.  ATE/ILS 

3.  Trained  Personnel 

a.  Manpower 

b.  Skills 

1.  Formal  Training 

2.  On-The-Job  Training 

4.  Diagnostic  Data  Collection/Analysis  System 

C.  D-Level  Diagnostics 

1.  Technical  Information 

a.  Maintenance  Aids 

b.  Paper-Based  Manuals 

c.  Diagnostic  Data  Base 

2.  Test  Equipment 

a.  Manual  Test  Equipment 

b.  Automatic  Test  Equipment 

1 .  ATE  Hardware 

2.  TPS 

3.  ATeiLS 
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3.  Trained  Personnel 

a.  Manpower 

b.  Skills 

1.  Formal  Training 

2.  On-The-Job  Training 

4.  Diagnostic  Data  Collection/Analysis  System 
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CHECKLIST 


Have  alternative  "diagnostic  subsystems"  been 
generated  for  optimization  purposes? 

E/  Does  the  "diagnostic  subsystem"  include  all  maintenance 
levels  as  well  as  program  management  and  engineering? 


Was  a  life  cycle  cost  model  used  that  was  sensitive 
to  diagnostic  parameters? 
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WEAPON 
SYSTEM 
ACQ.  PHASE 


WEAPON 

SYSTEM 

ACTIVITIES 


DIAGNOSTIC 

ACTIVITIES 


OPER.  CONCEPT  DEM/ 

REQMTS.  EXPLOR.  VAL 


PROD/ 

DEPL 


CONTRACT 

AWARD 


RISK 

ANALYSIS 


UPDATE 


DIAGNOSTIC  ACTIVITY 

The  initial  diagnostic  subsystem,  generated  to  implement  the  allocated 
diagnostic  requirements,  must  go  through  a  thorough  risk  analysis  during  the  Dem/Val 
Phase.  During  subsequent  Full-Scale  Development,  the  diagnostic  subsystem  is 
optimized  utilizing  results  of  trade  studies.  The  initial  diagnostic  element  mix  postulated 
during  Concept  Exploration  is  analyzed  by  the  contractor  for  risk  during  that  phase  by 
technology  assessments. 

PROCEDURE 

The  procedure  for  performing  risk  analysis  on  the  diagnostic  subsystem  will 
follow  the  same  type  of  assessments  conducted  for  risk  analysis  for  other  weapon  system 
elements.  For  example,  the  risk  assessment  for  a  radar  during  Dem/Val  will  include 
prototyping  of  its  new  development  components,  assessment  of  schedule,  cost  risks,  and 
assessment  of  the  overall  technologies  involved  in  the  development  and  integration  of  a 
total  system  to  meet  the  performance  requirements.  Since  the  diagnostic  subsystem  will 
be  treated  as  a  major  element  of  a  weapon  system,  the  same  procedures  should  apply  for 
it.  Heretofore,  diagnostic  subsystems  were  not  treated  as  an  entity  and  risk  analysis  was 
limited  only  to  the  physical  diagnostic  hardware,  such  as  automatic  test  equipment  and 
built-in  test. 

Risk  assessment  shall  include  the  isolation  within  the  diagnostic  subsystem  of 
all  development  and  non-development  items.  For  development  items,  weighting  factors  in 
terms  of  criticality  of  that  item  shall  be  assigned  and  the  items  shall  be  categorized  with 
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respect  to  risk.  For  items  considered  high  deveiopment  risk,  workarounds  shaii  be 
developed  and  trigger  points  for  decisions  on  their  implementation  shall  be  listed.  The  risk 
analysis  documentation  shall  be  utilized  to  impact  the  Statement  of  Work  for  the  DemA/al 
Phase.  During  Dem/Val  high-risk  items  shall  be  prototyped  and  demonstrated  to  the 
satisfaction  of  the  Government  Program  Manager. 

GUIDANCE 


The  Defense  Systems  Management  College  Program  Managers  Guide  contains 
an  extensive  chapter  (Chapter  12)  on  risk  assessment.  In  addition,  MIL-STD-1 388-1 
(Logistics  Support  Analysis)  contains  in  Tasks  203,  205,  and  303  guidance  on 
comparative  analysis,  supportability  related  design  factors,  and  evaluation  of  alternatives 
for  trade-off  analysis. 

Lessons  learned  have  pointed  to  some  overriding  areas  of  risk  which  must  be 
considered  during  the  initial  risk  analysis  assessment.  These  high-risk  areas  are  listed  in 
the  following  paragraphs: 

1 )  The  logistic  support  analysis  process  will  usually  generate  requirements  for 
each  of  the  logistic  elements  comprising  the  overall  logistic  system.  These  requirements 
are  based  upon  inputs  regarding  the  level  of  embedded  support  to  be  designed  into  the 
weapon  system.  The  Logistic  Element  Manager,  given  these  inputs,  proceeds  to  develop 
sparing  requirements,  support  equipment  requirements,  training  requirements,  etc.  A 
large  program  risk  area  occurs  when  the  promised  embedded  support  area  does  not 
materialize.  It  is  imperative,  therefore,  to  close  the  loop  between  assessment  during 
Dem/Val  of  the  diagnostic  element  capability  and  that  impact  on  all  logistic  elements. 

2)  A  second  major  risk  area  occurs  when  a  prime  weapon  system,  which  has 
been  developed  for  a  specific  maintenance  strategy  and  concept,  is  utilized  in  a 
completely  different  mission  environment.  For  example,  a  major  weapon  system  deployed 
in  a  three-level  maintenance  environment  may  be  required  to  operate  for  extended  periods 
of  time  in  a  dispersed  operating  location  with  less  than  full  support.  The  sustainability  and 
mobility  requirements  imposed  upon  that  weapon  system  may  not  have  been  included  with 
sufficient  priority  In  the  initial  analysis  to  develop  capability  for  that  operational 
environment.  It  is,  therefore,  imperative  that  as  part  of  the  risk  analysis,  the  assessment  of 
weapon  system  characteristics  and  the  application  of  weapon  system  priorities  be 
reviewed  prior  to  commitment  of  system  development  resources. 

3)  A  third  high  risk  area  worthy  of  special  consideration  is  the  analysis  of  the 
very  large  scale  integrated  circuits  and  very  high  speed  integrated  circuits  (VLSI/VHSIC). 
Despite  the  intensive  use  of  on-chip  testing  for  these  devices,  it  is  imperative  that  a 
standard  systems  approach  be  generated  by  the  contractor.  Testability  techniques 
including  signature  analysis  and  boundary  scan  designs  must  be  evaluated  at  the  system 
and  subsystem  level  prior  to  commitment  of  development  resources.  Standardization  by 
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the  contractor  of  the  embedded  support  architecture  will  eliminate  many  high-risk  problems 
caused  by  multiple  vendors  supplying  different  types  of  on-chip  testing. 

4)  A  fourth  high  risk  area  occurs  when  weapon  system  managers  fail  to 
comprehend  and  implement  the  existing  fielded  maintenance  standards  that  are  used  to 
support  the  deployed  system.  For  example,  the  military  has  for  many  years  been 
formalizing  the  use  of  lEEE-STD  716  C/ ATLAS  language  for  Depot  maintenance.  The 
GASS,  IFTE  and  MATE  programs  have  institutionalized  this  approach.  Despite  this  level 
of  standardization,  many  programs  completely  ignore  this  fact  during  the  Dem/Val  and 
FSD  Phases  of  a  program.  Since  the  targeted  Depot  ATE  has  been  standardized,  it  is 
possible  to  develop  test  programs  starting  with  Factory-level  testing  through  integration 
and  test  of  the  products  that  are  compatible  and  easily  translatable  to  the  fielded 
environment.  This  concept,  called  vertical  commonality,  will  mature  the  test  programs  so 
that  during  deployment  the  logistic  system  will  have  a  major  capability  and  remove  many 
of  the  risks  associated  with  transition  from  interim  contractor  support  to  full  government 
support.  Utilizing  expert  system  knowledge  during  these  same  phases  will  allow  the  test 
program  to  contain  levels  of  artificial  intelligence  to  extract  and  utilize  experience  data  on 
prior  failures  during  the  Deployment  Phase. 

5)  The  fifth  high  risk  area  is  the  incorporation  of  government  furnished 
equipment  (GFE)  in  weapon  systems.  Care  must  be  taken  to  ensure  that  the  diagnostic 
requirements  and  capability  are  known  and  verified.  The  Government  Program  Manager 
must  be  informed  if  the  required  weapon  system  diagnostic  capability  is  compromised  by 
deficiencies  in  the  GFE. 

6)  The  sixth  and  final  large  risk  area  Is  the  integration  and  test  of  the  weapon 
system  prior  to  delivery.  Since  weapon  systems  have  become  extremely  software 
dependent  and  since  many  weapon  systems  are  multi-mission  in  nature  utilizing  shared 
resources,  it  is  imperative  that  the  integration  and  test  function  in  a  program  be  utilized  to 
remove  as  much  risk  as  possible  from  the  weapon  system.  Integration  of  the  diagnostic 
elements  into  the  weapon  system  will  provide  a  major  "handle"  for  the  contractor  in  terms 
of  enhancing  the  integration  and  test  functions.  If  no  attention  is  paid  early  in  the  game  to 
this  high  potential  risk,  the  integration  and  test  functions  will  be  extremely  time 
consumming,  may  not  come  together  on  schedule,  and  may  cause  program  hardships.  If 
properly  achieved,  integration  and  test  can  be  streamlined  to  recover  much  of  the  upfront 
monies  spent  on  enhanced  testability  features.  It  is  therefore  imperative  that  this  area  be 
given  serious  attention  by  risk  assessment  studies  early  in  Concept  Exploration  and  again 
in  through  Dem/Val  and  Full-  Scale  Development. 
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CHECKLIST 

Was  risk  analysis  performed  for  the  entire 
"diagnostic  subsystem"? 

E/  During  DEM/VAL  were  testing/prototyping  efforts 
undertaken  to  determine  feasibility  of  achieving 
diagnostic  performance  requirements? 

O'  Were  adjustments  planned  for  in  those  cases  where 
one  of  the  diagnostic  elements  fails  to  meet 
expectations? 

Were  the  weapon  system  priorities  taken  seriously? 
E/  Have  the  integration  and  test  risks  been  defined? 
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DESIGNING  THE  DIAGNOSTIC  CAPABILITY 
OVERVIEW 

The  design  of  the  diagnostic  capability  is 
fractionated  among  a  number  of  system 
engineering  and  supportability  functions. 
Reliability,  maintainability,  integrated  logistic 
support,  testability,  human  engineering,  and 
safety  considerations  all  play  significant  roles  in 
determining  the  requirements  of  the  diagnostic 
capability  and  the  design  of  this  capability.  The 
design  process  is  further  fractionated  by  the 
relegation  of  this  capability  to  the  various  levels  of 
maintenance.  The  diagnostic  design  process  is 
controlled  by  a  large  number  of  military  standards 
which  deal  with  the  design  process  and  criteria. 
All  of  these  "pieces"  of  the  design  process  must 
not  only  work  together,  but  the  diagnostic  data 
produced  must  be  available  at  specific  times.  A 
break  in  one  of  the  links  can  compromise  the 
design.  A  cohesive,  integrated  design  process  is 
required.  It  is  the  Program  Manager’s  job  to  assure 
that  this  integration  is  realized  and  the  designer’s 
job  to  produce  this  effective  diagnostic  capability  in 
an  efficient  manner. 


IMPORTANT  CONSIDERATIONS  TO  BE  ADDRESSED 
Reamt. 

3.1  Promote  cohesiveness  and  efficiency  In  the  design  of  the  diagnostic 
capability. 


3.2  Establish  diagnostic  design  criteria  which  can  be  effectively  utilized. 
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WEAPON 

SYSTEM 

ACQ.  PHASE 

OPER. 

REQMTS. 

CONCEPT 

EXPLOR. 

DEM/ 

VAL 

FSD 

PROD/ 

DEPL. 

WEAPON 

SYSTEM 

ACTIVITIES 

AAA 

SDR  PDR  CDR 

DIAGNOSTIC 

ACTIVITIES 

7S.  A  A  A 

DIAGNOSTIC  PREUM.  DETAIL  FABRICATION 

SPEC.  DESIGN  DESIGN 

DIAGNOSTIC  ACTIVITY 


The  Government  Program  Management  function  has  limited  interaction  with  the 
design  process.  However,  it  would  be  unrealistic  to  expect  that  either  the  activities  leading 
up  to  the  design  phase  or  the  design  review  activity  can  be  performed  efficiently  without  a 
realization  of  the  potential  impact  these  activities  have  on  the  design  process. 

PROCEDURE 


The  cohesiveness  of  the  diagnostic  design  process  is  dependent  upon  the 
cohesiveness  of  the  design  information  flow.  Many  factors  impact  the  effectiveness  and 
efficiency  of  this  information  flow.  The  first  is  timing  -  What  is  done  and  in  what  sequence 
is  it  done?  The  second  factor  relates  to  the  various  disciplines  involved  in  the  design  of 
the  diagnostic  capability.  These  disciplines  are  controlled  by  a  sizeable  number  of  military 
standards,  which  relate  to  reliability,  maintainability,  testability,  safety,  human  engineering, 
software,  and  training.  These  standards  tend  to  fractionalize  the  design  of  the  diagnostic 
capability,  inasmuch  as  each  plays  a  significant  role  in  the  process.  The  third  factor  deals 
with  the  automation  of  the  design  process.  Computer-aided  tools  can  promote  the 
cohesiveness  and  the  efficiency  of  the  process.  Thus,  the  Government  Program  Manager 
must  understand  the  interfaces  among  these  various  engineering  and  logistic  functions  to 
assure  that  the  necessary  cohesiveness  is  achieved.  In  addition,  the  automation  of  the 
diagnostic  design  process  should  be  supported  and  encouraged  to  provide  a  more 
efficient  process  and  a  more  effective  diagnostic  capability. 
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GUIDANCE 


The  guidance  provided  in  this  section  is  designed  to  permit  maximum  visibility 
into  the  diagnostic  design  process.  The  Government  Program  Manager  cannot  simply  be 
satisfied  that  the  appropriate  requirements  have  been  levied  to  provide  the  desired 
diagnostic  capability.  To  avoid  shortfalls  In  the  fielded  system,  the  program  manager  must 
understand  the  design  process  flow,  timing,  and  data  requirements  which  must  be 
satisfied,  in  addition,  it  is  important  to  recognize  that  current  data  item  procurement 
practices  are  not  supportive  of  the  design  activity  in-process  data  needs.  Very  often,  the 
CDRL  and  associated  DID  do  not  adequately  reflect  these  in-process  needs.  The  high 
data  item  generation/revision  costs  generally  experienced  are  strong  motivators  for 
delaying  data  item  preparation  to  a  point  where  the  design  has  stabilized.  Such  motivation 
is  in  direct  conflict  with  the  utilization  of  the  data  to  make  design  decisions.  A  complete, 
detailed  data  submittal  indicating  that  the  design  is  flawed  is  of  little  use  after  the  design 
has  been  completed.  The  guidance  that  follows  has  been  designed  to  provide  the 
necessary  insight  into  the  design  process,  which  will  assist  the  Program  Manager  in  the 
progress  assessment  and  decision-making  process. 

Design  Environment 

The  diagnostic  design  environment  is  an  essential  component  of  the  overall 
diagnostic  design  activity,  which  has  been  established  by  the  contractor  in  response  to  the 
RFP  requirements.  This  environment  encompasses  both  the  implementation  methodology 
and  the  specialty  coordination  associated  with  the  diagnostic  design  process.  Evidence  of 
these  should  be  apparent  in  the  Interim  products  of  the  design  effort,  which  are  made 
available  to  the  government  program  management  function  (at  both  informal  in-process 
reviews  and  formal  system-level  design  reviews). 

Diagnostic  design  is  characterized  by  its  iterative  nature  and  a  high  degree  of 
interdependence  with  the  supportability  engineering  specialties  (i.  e.,  reliability, 
maintainability.  Integrated  logistic  support,  testability,  human  engineering,  and  safety). 
The  allocation  of  diagnostic  resources  must  be  based  on  inputs  from  these  disciplines. 
Therefore,  the  timing  and  quality  of  data  interchanges  must  be  in  accordance  with  the 
program  plans.  A  breakdown  in  data  availability  and  exchange  can  be  responsible  for 
program  delays  and  shortfalls  in  the  fielded  diagnostic  capability. 

The  data  flow  required  to  develop  the  composite  diagnostic  capability  must  be 
responsive  to  the  diagnostic  mix  established  for  the  specific  system  under  consideration. 
Embedded  diagnostic  features,  such  as  built-in  test  (BIT),  built-in  test  equipment  (BITE), 
system  integrated  test  (SIT),  performance  monitoring,  status  monitoring,  embedded 
training,  embedded  maintenance  aiding,  adaptive  Al-based  diagnostic  systems,  etc.,  are 
an  integral  part  of  the  prime  equipment  design.  Therefore,  the  diagnostic  data  flow 
associated  with  these  features  must  be  incremental  and  continue  until  the  detail  prime 
system  Configuration  Item  designs  are  complete.  For  the  external  diagnostic  elements, 
such  as  automatic  test  equipment  and  the  associated  test  program  sets,  manual  test 
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equipment,  portable  maintenance  aids,  technical  information  (hard  copy  or  electronic 
delivery),  training,  etc.,  the  diagnostic  data  flows  into  the  LSA  process  up  to  the  point 
where  the  firm  requirements  for  these  diagnostic  elements  can  be  established.  Once  firm 
requirements  exist,  the  diagnostic  design  environment  must  facilitate  a  smooth  transfer  of 
data,  which  is  sufficient  in  terms  of  detail  and  format  to  permit  fabrication  of  the  required 
external  diagnostic  capability. 

Program  management  must  develop  an  understanding  for  the  complexity  of  the 
data  flow  requirements  associated  with  the  program  under  consideration.  Given  the 
required  understanding,  maintaining  cognizance  over  the  content  and  timeliness  of  data 
availability  cannot  be  overemphasized. 

Table  3  is  a  listing  of  the  major  military  standards  which  influence  the  design  of 
the  diagnostic  capability.  Some  of  these  military  standards  are  programmatic  in  nature,  in 
that  they  establish  a  specific  program  and  describe  the  tasks  which  can  be  undertaken. 
The  remainder  of  the  standards  are  process  or  product-oriented.  As  can  be  seen,  these 
various  standards  influence  various  aspects  in  the  design  of  the  diagnostic  capability, 
starting  from  establishing  diagnostic  requirements,  through  the  design  and  assessment  of 
the  diagnostic  capability.  There  is  a  sequence  of  tasks  and  procedures  cited  in  these 
standards  which  can  be  applied  to  the  diagnostic  capability.  The  interfaces  and 
relationships  between  these  various  activities  are  complex  and  cannot  be  easily 
diagrammed  to  promote  understanding.  Establishing  diagnostic  requirements  is  described 
in  Requirement  #2,  and  the  assessment  is  described  under  Requirement  #4.  Thus  the 
following  guidance  will  address  the  design  of  the  embedded  and  external  diagnostic 
capability. 
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TABLE  3.  MILITARY  STANDARDS  APPUCABLE  TO  THE  DESIGN  OF  THE  DIAGNOSTIC  CAPABILITY 
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Design  Integration 

Figure  5  is  a  simplified  diagram  of  the  information  fiow  in  the  design  of  the 
diagnostic  capability.  The  design  process  begins  with  a  maintenance  concept  and  design 
data,  such  as  specifications,  biock  diagrams,  and  schematics.  Establishing  the  system’s 
architecture  is  the  next  step.  System’s  architecture  has  a  major  impact  on  the  design  of 
the  fielded  diagnostic  capabiiity.  The  concept  of  fault  tolerance  supports  the  maintenance 
concept  by  promoting  graceful  degradation  of  the  system’s  performance,  thereby  allowing 
the  maintenance  to  be  performed  at  the  user’s  convenience  rather  than  dictated  by  when 
faults  occur.  Design  for  testability  concepts  play  an  important  part  at  this  time.  Partitioning 
especiaiiy  is  closely  tied  to  fault  tolerance,  because  the  performance  monitoring  capability 
must  be  able  to  detect  failed  items  in  order  that  the  capability  of  the  system  is  known,  that 
necessary  switcning  to  alternate  means  is  facilitated,  and  that  maintenance  actions  can  be 
identified. 


The  Failure  Modes  and  Effects  Criticality  Analysis  (FMECA)  utilizes  the  system’s 
architecture  and  design  data  to  determine  the  modes,  causes  and  effects  of  item  failures. 
This  data  drives  the  maintenance  and  safety  requirements  which  in  turn  help  to  establish 
the  diagnostic  iogic,  test  point  seiection,  and  test  requirements.  From  this  information,  the 
diagnostic  capability  is  designed  and  fabricated,  including  the  testing,  (buiit-in  and 
external),  technical  information,  training,  and  personnel  capability.  Obviously  this  entire 
process  is  iterative  in  nature  •  a  factor  not  indicated  in  Figure  5. 

The  concept  of  vertical  testability  was  introduced  years  ago.  In  essence,  this 
concept  addressed  the  compatibility  of  testing  among  all  levels  of  maintenance,  including 
factory  testing.  The  core  of  this  concept  is  twofold.  The  first  is  the  establishment  of  a 
Cone  of  Tolerance  among  these  levels,  and  the  second  deals  with  the  compatibility  of 
environments  under  which  these  tests  are  performed. 
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i 


figure  5.  INFORMATION  FLOW  IN  THE  DESIGN  OF  THE  DUGNOSTIC  CAPABILITY 
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TOLERANCE 
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RGURE  6.  CONE  OF  TOLERANCE 


The  Cone  of  Tolerance  concept  Is  depicted  in  Figure  6,  in  which  the  testing 
tolerances  are  widened  as  the  unit  is  tested  closer  to  its  operational  environment.  The 
compatibility  of  testing  environments  can  be  implemented  best  through  the  use  of  common 
test  equipment  at  Intermediate,  Depot,  and  Factory  Levels. 

Extension  of  this  vertical  testability  concept  is  recommended  for  the  entire 
fielded  diagnostic  capability.  Figure  7  depicts  this  concept,  in  which  vertical  testing  is 
shown  on  the  left  and  is  joined  by  technical  information  and  personnel  and  training 
compatibility  requirements.  Not  only  is  this  compatibility  required  vertically,  but  also 
horizontally.  All  elements  that  make  up  the  diagnostic  capability  must  be  compatible  at 
each  maintenance  level. 

This  concept  of  vertical  and  horizontal  compatibility  is  key  to  the  integration  of 
diagnostic  capability.  The  entire  process  is  driven  by  the  diagnostic  logic  which  effects  the 
requirements  for  all  of  the  diagnostic  elements.  This  diagnostic  logic  can  be  established 
by  a  variety  of  means  including  the  use  of  maintenance  dependency  charts,  fault  trees. 
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etc.  To  implement  this  concept,  a  series  of  matrices  similar  in  format  to  Figure  7  can  be 
prepared  at  various  system  hierarchy  levels  (e.g.,  system,  subsystem,  LRU,  SRU).  These 
matrices  should  be  tailored  to  the  unique  requirements  of  a  specific  weapon  system  and 
may  be  used  in  conjunction  with  other  required  data  deliverables  (e.g.,  test  requirements 
document). 
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RGURE  7.  DIAGNOSTIC  DESIGN  INTEGRATION 
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AUTOMATION  OF  THE  DIAGNOSTIC  DESIGN  PROCESS 

Automation  of  the  diagnostic  design  process  is  encouraged  to  provide  a  more 
efficient  and  effective  design  process.  The  diagnostic  design  process  should  be  an 
integral  part  of  prime  system  computer-aided  engineering  and  design. 

The  added  efficiency  and  effectiveness  in  the  use  of  automation  is  reflected  in  a 
number  of  ways.  The  effect  of  changes  in  either  the  diagnostic  design  or  the  prime  system 
design  can  be  readily  ascertained  as  the  design  progresses.  This  iterative  process  then 
can  give  the  Contractor  Program  Manager,  as  well  as  the  designer,  information  on 
whether  or  not  the  diagnostic  specification  requirements  will  be  met.  In  addition, 
automation  permits  the  concurrent  development  and  evaluation  of  the  entire  diagnostic 
capability  along  with  the  remainder  of  the  prime  system. 

The  diagnostic  design  and  assessment  tools  enhance  the  effectiveness  and 
efficiency  of  the  process.  A  description  of  available  tools  and  processes  is  available  in  the 
Design  Encyclopedia  Guide. 
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CHECKLIST 


Has  a  diagnostic  design  environment  been  adequately 
defined  and  imposed  to  ensure  that  diagnostic  design 
requirements  are  considered  as  part  of  the  mainstream 
design  effort? 

Has  a  concerted  effort  been  made  to  assure  vertical 

and  horizontal  integration  and  compatibili^  of 

all  elements  which  comprise  the  diagnostic  capability? 

Has  this  been  documented  for  review? 

Of  Have  steps  been  taken  to  utilize  automotion  of  the 
dioanostic  design  process  to  enhance  design  efficiency 
and  to  improve  the  effectiveness  of  the  fielded 
diagnostic  capabilii^? 

of  What  measures  will  be  taken  to  ensure  that  vertical 
diagnostics  is  being  implemented? 

of  How  are  the  interfaces  with  training,  technical  infor¬ 
mation,  and  off-line  test  being  managed  to  facilitate 
concurrent  delivery  of  weapon  system  and  support  to  the 
extent  required  for  test  and  evaluation  and  maturation? 
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REQUIREMENT  #3.2 


OPER.  CONCEPT 

ACO^E 


WEAPON 

SYSTEM 

ACTIVITY 


DEM/ 

VAL 


PROD/ 

DEPL 


SYSTEM 

ANALYSES 


SYS.  PREL.  DETAIL 
SPEC.  DESIGN  DESIGN 


DIAGNOSTIC 

ACTIVITIES 


DIAGNOSTIC  ACTIVITY 

Design  of  the  diagnostic  capability  and  the  elements  that  make  up  this  capability 
are  initiated  early  in  weapon  system  development.  It  begins  soon  after  initial  analyses  and 
allocation  are  completed  and  extends  at  least  until  the  end  of  Full-Scale  Development. 
Design  criteria  and  guidance  need  to  be  avaiiable  for  use  as  the  diagnostic  capability 
design  progresses.  Obviously,  the  bulk  of  this  design  guidance  is  utilized  by  the  designer 
of  the  prime  system  and  its  support  capability.  The  Government  Program  Manager  needs 
to  be  aware  of  the  type  of  guidance  that  is  availabie  and  if  the  contract  specifies  any 
design  criteria. 

PROCEDURE 

Design  criteria  and  guidance  relating  to  the  diagnostic  capability  and  individual 
diagnostic  elements  are  available  from  a  number  of  sources,  including  standards, 
handbooks,  and  guides.  Most  often,  this  guidance  is  not  a  contractual  requirement, 
except  when  a  specific  military  standard  is  invoked.  However,  for  the  most  part,  the 
contractor  should  utilize  this  guidance  material  as  he  sees  fit,  as  long  as  diagnostic 
requirements  are  met  and  interfaces  are  controlled. 

Guidance  to  the  Government  Program  Manager  consists  of  identifying 
applicable  guidance  documents,  and  where  published  material  is  not  readily  available 
limited  guidance  is  furnished. 
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GUIDANCE 


Detailed  design  criteria  and  guidance  in  relation  to  the  entire  diagnostic 
capability,  as  well  as  for  each  diagnostic  element,  are  addressed  in  detail  in  the  Design 
Encyclopedia  Guide.  A  substantial  amount  of  information  that  is  contained  in  that  guide  is 
not  available  in  other  existing  guides.  Guidance  contained  in  this  Government  Program 
Managers  Guide  is  limited  to  references  to  other  available  design  criteria  and  guidance. 

The  following  are  references  to  existing  design  criteria  and  guidance. 

General  Guidance 

1 .  MIL-STD-454,  Standard  General  Requirements  for  Electronic  Equipment 

This  standard  covers  the  common  requirements  to  be  used  in  military 
specifications  for  electronic  equipment.  Reliability,  maintainability,  and 
human  engineering  requirements  are  included  in  this  standard.  However,  for 
these  types  of  engineering  disciplines,  the  guidance  stresses  that  this 
standard  does  not  establish  requirements  and  must  not  be  referenced  in 
contractual  documents.  These  three  requirement  examples  offer  direction 
on  what  should  be  considered  in  preparing  contractual  documents. 

2.  MIL-STD-41 5,  Design  Criteria  for  Test  Provisions  for  Electronic  Systems  and 
Associated  Equipment 

This  standard  establishes  design  criteria  for  test  provisions  that  permit  the 
functional  and  static  parameters  of  electronic  systems  and  associated 
equipment  to  be  monitored,  evaluated,  or  isolated.  The  standard,  in  its 
present  form  (Revision  D),  addresses  older  technologies  and  thus,  if 
referenced  in  contractual  documents,  must  be  tailored  to  address  only 
certain  provisions  in  this  standard. 

3.  The  RADC  Reliability  Engineers  Tool  Kit 

The  tool  kit  is  intended  for  use  by  a  practicing  Reliability  and  Maintainability 
(R&M)  engineer.  Emphasis  is  placed  on  his  role  in  the  various  R&M 
activities  of  an  electronic  systems  development  program.  The  tool  kit  is  a 
compendium  of  useful  R&M  reference  information  to  be  used  in  everyday 
practice. 

System  Architecture 

There  are  a  number  of  guides,  which  address  the  architecture  of  the  system 
design,  that  promote  improvements  in  the  system’s  diagnostic  capability.  Included  are: 
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1.  Architecture  Specification  for  PAVE  PILLAR  Avionics,  January  1987, 
SPA90099001A 

This  specification  addresses  the  advanced  avionics  architecture  which  is 
specifically  targeted  for  advanced  tactical  fighters  and,  in  general,  for  all 
military  aircraft  applications.  This  architecture  promotes  a  much-improved 
approach,  which  will  foster  an  improved  diagnostic  capability.  An  example  of 
this  approach  is  contained  in  the  Design  Encyclopedia  Guide. 

2.  Reliability,  Testability  Design  Considerations  for  Fault  Tolerant  Systems 
(RADC-TR-84-57) ,  Apr  84,  Limited  to  USQO  agencies  &  their  contractors; 
critical  technology. 

Furnishes  reliability  and  testability  evaluation  and  application  guidance  for 
fault-tolerant  designs. 

3.  RADC  Testability  Notebook,  Final  Technical  Report,  June  1982 

This  notebook  presents  a  consolidation  of  Information  relating  to  testability 
design  techniques,  procedures,  cost  trade-off  tools,  and  the  relationship  of 
testability  to  other  design  disciplines  and  requirements.  Specific  examples  of 
good  testability  design  are  contained  in  this  document. 

4.  Avionics  Testability  Design  Guide  -  MATE  Guide  3,  Part  Three,  Testability 
Design  Handbook 

This  portion  of  the  MATE  Guides  discusses  testability  design  techniques. 

5.  MIL-STD-2165,  Testability  Program  for  Electronic  Systems  and  Equipments 

Appendix  B  of  MIL-STD-2165  cites  a  series  of  factors  which  affect  the 
inherent  testability  of  a  weapon  system.  This  information  can  be  used  either 
as  design  guidance  or,  if  weighted  and  scored,  can  actually  provide  a  Figure 
of  Merit  for  a  specific  system/unit. 

6.  Testability  Analysis  Handbook  (Draft) 

This  handbook  is  presently  in  draft  form.  This  handbook  will  serve  as  a 
vehicle  for  implementing  MIL-STD-2165. 


Built-In  Test 

1 .  Built-In  Test  Design  Guide-Joint  AMC/CNO/AFLC/AFSC  Commanders,  April 
1987,  Subject  to  Export  Control  Laxvs. 
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This  Joint  Service  BIT  Design  Guide  provides  detailed  guidelines  on  the 
implementation  of  BIT,  including  BIT  design  techniques  at  all  levels  within 
the  weapon  system. 


2.  Smart  BIT  (RADC-TR-85-1 48) ,  Aug  85 ,  Limited  to  USGO  agencies  &  their 
contractors;  critical  technology. 

Applications  of  artificial  intelligence  techniques  in  the  design  of  BIT,  to 
minimize  false  alarms,  retest  OKs  and  non-required  maintenance. 

3.  Sensor  Handbook  for  Test,  Monitoring,  Diagnostic,  and  Control  System 
Applications  to  Military  Vehicles  and  Machinery,  National  Bureau  of 
Standards 

This  handbook  is  Intended  as  a  guide  for  those  who  design,  specify,  use, 
and  test  weapon  systems  containing  monitoring  sensors.  The  handbook 
addresses  measures  and  principles  of  measurement,  data  acquisition, 
sensor  calibration  and  testing,  environmental  considerations,  stability, 
durability,  reliability,  and  error  assessment  for  various  types  of  sensors. 

Automatic  Test  Equipment  (ATE) 

1 .  Modular  Automatic  Test  Equipment  (MATE)  Guides 

Although  Air  Force-oriented,  these  guides  describe  procedures  and 
techniques  for  acquiring  automatic  test  equipment.  Of  particular  interest  is 
Guide  5,  which  addresses  the  acquisition  of  test  program  sets. 

2.  MIL-STD-2077,  General  Requirements,  Test  Program  Sets 

This  standard  covers  the  acquisition  of  test  program  sets  for  use  with  ATE. 
Design  criteria  is  included,  which  addresses  many  detail  requirements  for 
TPSs. 

Human  Engineering 

1.  MIL-STD-1472,  Human  Engineering  Design  Criteria  for  Military  Systems, 
Equipment,  and  Facilities 

This  standard  covers  general  human  engineering  design  criteria  which  can 
be  applied  to  any  weapon  system. 
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Technical  Information 

There  are  a  variety  of  standards  which  address  the  preparation  of  technical 
publications.  Most  of  these  documents  are  directed  at  a  specific  military  service.  All 
address  the  delivery  of  paper-type  documentation.  There  is  no  firm  guidance  relating  to 
innovative  means  for  generating  and  delivering  technical  information.  In  the  past,  this  has 
caused  expensive,  voluminous,  inaccurate,  and  untimely  technical  publications  to  be 
delivered.  These  deficiencies  can  best  be  described  in  the  DoD  Audit  Report  No.  87-115, 
April  3,  1987,  "Summary  Report  on  the  Defense-Wide  Audit  on  Acquisition  of  Technical 
Manuals  and  Related  Data  From  Contractors." 

Means  should  be  sought  for  generating  and  delivering  this  technical  information 
in  a  less  costly  manner,  without  compromising  its  quality.  There  are  a  number  of  tools 
available,  or  under  development,  which  can  assist  the  designer  of  this  technical 
information  in  authoring  the  text,  when  electronic  delivery  of  technical  information  is 
contemplated.  Some  guidelines  and  standards  for  automatic  generation  of  technical 
information  and  its  delivery  electronically  can  be  obtained  from  the  Human  Resources 
Laboratory  at  Wright-Patterson  Air  Force  Base.  This  guidance  information  has  been 
developed  under  the  Integrated  Maintenance  Information  System  (IMIS)  Program. 

Innovative  ideas  for  displaying  this  technical  information  are  encouraged,  as 
stipulated  in  Task  303,  MIL-STD-1 388-1.  Care  should  be  taken  to  provide  for  quick 
access  to  the  required  data.  For  electronic  delivery  of  this  data,  formats  may  vary 
substantially  from  paper-based  technical  manuals.  Previously  specified  access  times  and 
information  modification  times  should  influence  the  type  of  generation  and  delivery 
methods.  DoD-lnstruction  4151.9  requires  the  services  to  plan  and  schedule  the 
acquisition  of  technical  manuals  (technical  information)  to  ensure  their  availability  in  final 
form  before,  or  concurrently  with,  delivery  of  the  system  to  the  field.  During  design,  final 
plans  should  be  developed,  along  with  the  support  equipment  which  is  furnished. 

Maintenance  Aids 

There  is  a  need  to  present  technical  information  and  troubleshooting  advice  to 
the  technician  on  location  and  readily  available  for  his  use.  The  maintenance  aid, 
sometimes  called  a  job  performance  aid,  presents  information  generated  by  experts  to 
assist  the  less-experienced  technician. 

The  maintenance  aid  is  a  device,  publication,  or  guide  used  on  the  job  to 
facilitate  performance  of  maintenance.  It  delivers: 

o  Historical  information  on  what  fault  was  found  when  similar  symptoms  were 
experienced 

o  Troubleshooting  logic  to  assist  in  finding  the  fault 
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o  Procedural  information  which  assists  the  technician  in  finding  and  correcting 
a  failure. 

Normally,  a  maintenance  aid  is  used  in  conjunction  with  a  testing  capability. 
Maintenance  aids  could  be  paper-based  or  employ  electronic  delivery  systems. 

Electronic  delivery  of  this  type  of  information  opens  the  door  to  solving  some  of 
the  problems  associated  with  paper  maintenance  aids.  Two  attributes  of  electronic 
delivery  systems  are: 

o  Information  can  be  available  to  the  technician  in  a  matter  of  seconds  by 
carefully  constructed  menus,  in  lieu  of  the  technician  having  to  page  through 
a  paper  document. 

o  The  collection  of  historical  data  and  the  subsequent  modification  to  the 
software  programs  which  deliver  this  technical  information  can  be  updated 
in  a  matter  of  seconds,  instead  of  a  matter  of  months. 

This  latter  attribute  lends  itself  to  the  introduction  of  expert  systems,  which  often 
employ  artificial  intelligence  technology.  The  expert  system  has  the  capability  of 
combining  various  pieces  of  information  to  lead  the  technician  to  a  logical  decision  on  what 
is  faulty  and  how  it  can  be  repaired. 

Another  important  aspect  of  the  maintenance  aid  is  its  ability  to  train  technicians 
on  the  job.  Training  programs  must  be  closely  associated  with  the  design  and 
development  of  a  maintenance  aid. 

Over  the  past  20  years,  many  maintenance  aids  have  been  designed, 
developed,  and  tested  in  the  field,  mostly  under  laboratory-controlled  conditions.  These 
tests,  for  the  most  part,  have  proven  successful.  However,  the  transition  of  these 
maintenance  aids  Into  the  field  has  not  been  accomplished  to  any  great  extent.  One  of  the 
reasons  Is  that  specifications,  standards,  and  guidance  information  on  how  to  Invoke  this 
requirement  are  lacking. 

A  few  important  facts  should  be  remembered  when  applying  maintenance  aids 
and  expert  systems. 

o  The  design  of  the  maintenance  aids  must  be  done  concurrently  with  the 
user.  Once  a  working  model  of  the  equipment  is  available,  there  should  be 
a  dynamic  interchange  of  information  between  the  maintenance  technician 
and  the  design  engineer  to  ensure  an  effective  and  efficient  man-machine 
interface  is  attained. 
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o  User  acceptance  and  adoption  of  maintenance  aids  will  be  facilitated  in 
cases  where  potential  users  are  given  a  trial  period  in  which  to  become 
familiar  with  these  devices  prior  to  their  formal  implementation. 

o  A  system  must  be  devised  to  assure  timely  updating  of  information  to 
correct  errors  and  to  add  newly  acquired  information.  Without  such  a 
system,  the  maintenance  aid  will  quite  rapidly  become  obsolete. 

Manpower  and  Training 

After  personnel  and  training  requirements/allocations  have  been  made,  the 
training  curriculum  needs  to  be  established  concurrently  with  the  system  detail  design. 
This  includes  the  formal  schooling  curriculum,  as  well  as  on-the-job  training.  One  of  the 
alternatives  available,  if  electronic  delivery  of  technical  information  is  employed,  is 
combining  training  aids  with  the  delivered  technical  information.  These  two  types  of 
information  (aiding  and  training)  are  somewhat  similar  in  nature  and,  at  times, 
indistinguishable.  The  training  curriculum  should  be  aimed  at  the  user(s)  and  accessed  in 
a  manner  which  can  be  utilized  by  a  variety  of  users. 

These  training  devices  can  be  freestanding  or  embedded  in  the  prime  system. 
Separate  and  distinct  training  devices  (maintenance  trainers)  may  be  required  to  be 
developed  for  the  formal  schooling. 

Integration  of  Diagnostic  Elements 

There  are  a  variety  of  ways  in  which  diagnostic  elements  can  be  integrated  to 
produce  a  more  effective  and  efficient  diagnostic  capability.  Expert  system  technology  can 
bo  incorporated  in  either  ATE  or  BIT  to  supplement  the  basic  testing  capability.  Fault- 
tolerant  design  and  testability  design  can  be  introduced  into  prime  system  architecture  to 
promote  ease  of  testing,  along  with  graceful  degradation.  Maintenance  aiding  and 
maintenance  training  can  be  combined  to  provide  on-the-job  maintenance  and  training 
information,  utilizing  a  single  device  or  embedded  into  the  prime  system.  Several 
comprehensive  examples  of  this  integration  appear  in  the  Design  Encyclopedia  Guide. 


3-21 


CRITERIA  FOR  DESIGNING 
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REQUIREMENT  #3.2 


CHECKLIST 


£3"  Is  the  contractor  utilizing  available  design  guidance? 

E?  Is  the  contractor  attempting  to  integrate  the  various 
diagnostic  elements  to  provide  a  more  effective  and 
efficient  diagnostic  capability? 


ASSESSMENT  REQUIREMENT  #4 


ANALYSIS  AND  ASSESSMENT  OF  THE  PERFORMANCE  OF  THE  DIAGNOSTIC 
CAPABILITY 


OVERVIEW 

Throughout  the  design  of  the  weapon  system’s 
diagnostic  capability,  it  is  essential  to  analyze, 
assess,  and  demonstrate  its  performance.  Such 
assessments  are  an  integral  part  of  logistics, 
reliability,  maintainabiiity,  testability,  human 
engineering,  software  and  safety  programs.  The 
ability  to  properly  conduct  these  analyses, 
assessments,  and  demonstrations  is  hampered  by 
the  lack  of  available  techniques  and  tools  to  help, 
and  the  incompatibility  of  the  available  tools  and 
techniques  to  function  together.  Thus  both  the 
program  manager  and  the  designer  must  have 
sufficient  knowledge  to  understand  the  processes 
utilized  and  integrate  these  processes  and  tools  to 
do  the  best  possible  job. 


TESTABILITY/ 

DIAGNOSTICS 


PROGRAMMATIC 


] 


^  REQUIREMENTS  | 


DESIGN 


—  H  ASSESsS^j 

■  j  REVIEWS  I 

—  EVALUATION  | 

■  I  MATURATION  | 


IMPORTANT  CONSIDERATIONS  TO  BE  ADDRESSED 

Reqmt. 

4.1  Analysis  and  assessment  of  the  diagnostic  capability  should  be 
performed  for  the  entire  diagnostic  capability,  as  well  as  for  each 
diagnostic  element 

4.2  Maintainability  demonstrations  should  be  designed  to  verify  that 
diagnostic  requirements  have  been  met 
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WEAPON 

SYSTEM 

ACQ.  PHASE 

OPER. 

REQMTS. 

CONCEPT 

EXPLOR. 

DEM/ 

VAL 

FSD 

PROD/ 

DEPL. 

WEAPON 

SYSTEM 

ACTIVITIES 

AAA 

SYSTEM  PREL.  DETAIL 

SPEC.  DESIGN  DESIGN 

DIAGNOSTIC 

ACTIVITIES 

A  A  A 

IN-PROCESS  ASSESSMENTS 

DIAGNOSTIC  ACTIVITY 

During  Dem/Val  and  FSD,  it  is  important  to  assess  whether  the 
testability/diagnostic  requirements  are  being  achieved.  This  activity  encompasses  all 
preliminary  and  full-scale  engineering  activities  pertaining  to  both  the  embedded  and 
external  diagnostic  capability. 

PROCEDURE 


In-process  testability/diagnostic  analyses  can  be  conducted  at  most  any  time 
within  Dem/Val  and  FSD.  These  in-process  analyses  are  typically  reviewed  by  the 
government  at  preliminary  design  reviews  and  critical  design  reviews.  These  analyses 
are,  for  the  most  part,  implemented  per  MIL-STD-2165  (Task  202,  Preliminary  Design,  and 
Task  203,  Detail  Design).  Normally,  these  analyses  will  be  the  responsibility  of  the  design 
engineer.  However,  it  remains  the  job  of  the  Government  and  Contractor  Program 
Managers  to  understand  the  processes  utilized  and  to  interpret  the  results  of  these 
analyses. 
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GUIDANCE 


Basically,  there  are  two  types  of  in-process  analyses.  The  first  deals  with  the 
inherent  testability  of  the  hardware  design  and  is  independent  of  test  stimuli  and  response 
data.  The  second  type  deals  with  the  effectiveness  of  the  diagnostic  capability  which 
deals  with  measures  that  include  consideration  of  hardware  design,  embedded 
diagnostics,  and  external  diagnostics.  Diagnostic  effectiveness  measures  include,  but  are 
not  limited  to,  fault  coverage,  fault  resolution,  fault  detection  time,  fault  isolation  time,  and 
false  alarm  rate. 

There  are  a  number  of  tools  available,  both  automatic  and  manual,  which  can 
be  used  to  assist  in  these  analyses.  Descriptions  of  these  tools  and  their  utilization  are 
contained  in  the  Diagnostic  Design  Encyclopeida.  However,  the  Contractor  Program 
Manager  should  understand,  at  minimum,  the  types  of  tools  ;which  fall  into  these  two 
categories. 

The  first  type  of  tool  deals  with  the  inherent  testability.  These  tools  are 
especially  important  during  preliminary  design,  where  system  block  diagrams,  system 
architecture  diagrams,  system  busing,  and  test  point  schemes  are  defined.  Two  varieties 
of  these  types  of  tools  exist.  The  first  deals  with  control  and  observation  of  the  states  of 
internal  nodes  within  a  digital  circuit.  These  tools  generally  calculate 
Controllability/Observability  Figures  of  Merit.  This  type  of  analysis  supports  decisions  to  fix 
hard-to-control  and  hard-to-observe  nodes  by  adding  test  points  to  the  digital  hardware.  A 
more  accurate  analysis  may  be  achieved  using  the  Dependency  Model  approach,  which 
makes  use  of  the  test  strategy  information,  as  well  as  circuit  topological  information. 
These  tools  apply  to  digital,  analog,  and  nonelectronic  systems.  In  addition  to  providing 
static  Testability  Figures  of  Merit,  such  as  average  Inherent  ambiguity  group  size  and 
feedback  loop  characteristics,  they  can  provide  dynamic  Figures  of  Merit  such  as  mean 
time  to  fault  isolate. 

The  second  type,  diagnostic  effectiveness  tools,  are  characterized  by  fault 
simulators  and  BIT  effectiveness  analyzes.  For  digital  circuits,  the  most  accurate  analysis 
may  be  achieved  using  fault  simulation,  which  directly  yields  FD/FI  predictions.  Actual  test 
factors  are  used  to  derive  the  simulation  software.  Simulation  run  time  and  costs 
essentially  preclude  the  use  of  a  fault  simulator  to  make  "what  if  iterations  on  different  test 
point  placement  combinations.  Fault  simulation  methods  may  also  be  applied  for  deriving 
measures  of  BIT  effectiveness.  BIT  effectiveness  analysis  consists  more  of  an  accounting 
process,  to  keep  track  of  which  functions,  modules,  and  interconnections  are  covered  by 
which  built-in  tests. 
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IN-PROCESS  TESTABIUTY/DIAGNOSnC  ANALYSIS 


REQUIREMENT  #4.1 


CHECKUST 


Does  the  analysis  of  testabilify/diagnostic  requirements 
address  all  major  support  disciplines? 

Off-line  ATE 
Embedded  diagnostics 

Manpower  required  to  support  analysis  outputs 
Training  requirements 
Information  requirements. 

O'  Are  all  analyses  complete  and  unambiguous? 

Do  they  meet  specification  requirements? 

Is  the  analysis  integrated  and  cohesive?  Are  any 
requirements  in  conflict? 

O'  Are  the  training,  information,  and  manpower  require¬ 
ments  adequately  scoped  and  specified  to  support  the 
technical  complexity  of  the  subject  end  item  m  its 
operational  environment? 


What  design  automation  tools  (e.g.  simulators, 
analyzers)  has  the  contractor  proposed/used  to 


MAINTAINABILITY  DEMONSTRATIONS 


REQUIREMENT  #4.2 


WEAPON 
SYSTEM 
ACQ.  PHASE 


OPER. 

REQMTS. 


CONCEPT 

EXPLOR. 


DEM/ 

VAL 


FSD  P^ROD/ 


WEAPON 

SYSTEM 

ACTIVITIES 


DIAGNOSTIC 

ACTIVITIES 


DIAGNOSTIC  ACTIVITY 

Maintainability  Demonstrations  are  performed  in  accordance  with  the 
appropriate  demonstration  method  contained  in  MIL-STD-471A.  Notice  2  of  MIL-STD- 
471 A  contains  requirements  for  demonstration  and  evaluation  of  system  BIT/external 
tesVfault  isolation/testability  attributes.  This  method  will  demonstrate  the  integration  of  the 
diagnostic  capability  for  the  system  (e.g.,  integration  of  embedded  test  software  and 
hardware  techniques,  automatic  and  manual  test,  BIT/SIT,  training  levels,  human 
interfaces).  The  Maintainability  Demonstrations  evaluate  the  diagnostic  performance  of 
the  system  with  respect  to  the  diagnostic  performance  criteria  and  objectives  established 
In  accordance  with  MIL-STD-470  (Maintainability  Program)  and  MH.-STD-2165  (Testability 
Program)  and  the  requirements  for  an  "integrated”  diagnostics  capability  demonstration 
contained  in  the  FSD  SOW. 


PROCEDURE 

The  integrated  diagnostics  process  increases  the  scope  of  maintainability 
demonstrations.  It  is  the  Government  Program  Manager’s  responsibility  to  ensure  that  this 
increased  scope  is  understood  and  implemented. 

The  scope  of  Maintainability  Demonstrations  includes: 


1 .  Demonstration  of  Testability  Parameters 


MAINTAINABIU7Y  DEMONSTRATIONS 


REQUIREMENT  #4.2 


-  BIT  Fault  Detection 

-  BIT  Isolation  Time 

-  BIT  Fault  Isolation  Level  (Ambiguity  Group) 

2.  Demonstration  of  Test  Effectiveness  (ATE)  (MIL-STD-2077) 

-  ATE  Fault  Detection 

-  ATE  Fault  Isolation  Time 

-  ATE  Fault  Isolation  Level  (Ambiguity  Group) 

-  UUT/ATE  Compatibility 

3.  Demonstration  of  Technical  Information 

-  Technical  Information  Access  Time 

-  Technical  Information  Relative  Access  Ease 

-  Technical  Information  Format 

-  Technical  Information  Usability 

4.  Demonstration  of  Training/Skills 

-  Relationship  between  maintenance  procedures  and  skills 

-  Relationship  between  formal  training  and  actual  maintenance 
job  flow. 

5.  Demonstration  of  Vortical  and  Horizontal  Integration 

-  Compatibility  and  Consistency  of  diagnostic  demonstration  results 
between  maintenance  levels  and  among  their  respective  diagnostic 
olements. 

GUIDANCE 


Unfortunately,  the  ability  to  carry  out  a  single  demonstration,  or  even  a 
series  of  demonstrations,  to  prove/evaluate  this  level  of  diagnostic  capability  is 
dependent  upon  having  all  of  the  diagnostic  elements  available  for  the 
maintainability  demonstration.  While  this  should  always  be  the  goal,  it  may  not  be 
feasible  due  to  development  schedules,  UUT  design  instability,  data  availability  and 
other  overall  program  constraints.  (Note  that  this  is  a  primary  reason  for  a 
Diagnostics  Maturation  Program.) 

The  Government  Program  Manager  must  determine,  based  upon  specific 
program  characteristics  and  constraints,  an  appropriate  scope  of  the  Maintainability 
Demonstrations.  This  scope  must  then  be  made  a  contractual  requirement. 
Demonstration  procedures  and  sample  sizes  must  be  defined  and  clear  delineation 
of  responsibilities  must  be  established.  Typically,  the  contractor  prepares  a 
Maintainability  Demonstration  Plan  early  in  the  FSD  Phase  and  that  plan  is  subject  to 
government  review  and  approval.  The  Government  Program  Manager  should  take 
advantage  of  this  opportunity  to  maximize  the  scope  and  effectiveness  of  the 
Maintainability  Demonstrations.  The  ability  to  maximize  the  scope  and  effectiveness 
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of  the  Maintainability  Demonstrations  will  have  a  significant  cost-savings  impact  on 
the  Diagnostics  Maturation  Program  requirements.  Maintainability  Demonstrations 
represent  the  first  major  opportunity  to  evaluate  the  level  of  diagnostic  capability 
achieved.  Also,  Maintainability  Demonstrations  can  be  conducted  early  enough  to 
implement  corrective  action  cost-effectively.  Demonstrations  are  conducted  while 
the  system  is  still  considered  to  be  in  the  Development  Phase.  After  the 
demonstrations  are  completed,  the  relative  cost  of  identifying  deficiencies  and 
implementing  corrective  action  is  significantiy  increased.  A  significant  milestone  of 
'Government  Acceptance’  occurs  upon  satisfactory  completion  of  Maintainability 
Demonstrations.  After  this  milestone,  costs  for  identification  and  resolution  of 
diagnostic  deficiencies  may  be  subject  to  contract  interpretation  and/or  negotiation. 

The  total  strategy  for  the  test  and  evaluation  of  the  diagnostic  capability  is  placed  on 
the  TEMP,  and  detailed  in  the  integrated  Test  Plan. 

Based  upon  the  selected  scope  of  the  Maintainability  Demonstration, 
procedures  from  MIL-STD-471  are  utilized  and  adapted  for  the  scope.  These 
procedures  are  documented  in  the  Maintainability  Demonstration  Plan,  and 
therefore,  the  Data  Item  Description  for  the  Maintainability  Demonstration  Plan  must 
be  modified  accordingly.  The  results  of  the  Maintainability  Demonstration  are 
documented  in  a  technical  report  -  Maintainability  Demonstration  Results. 

Concurrent  Demonstrations 

The  overall  diagnostic  capability  is  the  sum  of  a  variety  of  diagnostic 
elements.  Therefore,  a  requirement  should  be  established  for  early  demonstration 
of  the  entire  diagnostic  capability  produced  by  the  integration  of  all  of  these 
diagnostic  elements.  This  is  referred  to  as  concurrent  demonstrations,  where  the 
timing  of  various  diagnostic  element  demonstrations  are  planned  and  scheduled  for 
concurrency  so  that  the  integrated  capability  can  be  assessed. 

The  Government  Program  Manager  must  establish  the  requirement  or 
goal  for  concurrent  demonstrations.  Development  schedules  must  be  evaluated 
early  to  determine  the  level  of  concurrency  feasible.  Critical  and/or  risk  areas  must 
be  identified  and  evaluated. 

For  each  element  of  the  diagnostic  capability,  demonstration  requirements 
are  established.  The  diagnostic  capability  factors  to  be  demonstrated,  as  the  result 
of  the  combining  or  integration  of  the  elements,  must  be  specifically  identified.  For 
example,  a  demonstration  of  subsystem  BIT  may  prove  fault  detection  and  isolation 
levels.  A  demonstration  of  ATE  may  prove  external  fault  detection  and  isolation 
levels.  A  concurrent  demonstration  of  these  two  diagnostic  elements  will  prove  the 
ability  of  the  ATE  to  use  BIT  circuitry,  to  use  BIT  results,  and  the  consistency  of  test 
results  between  BIT  and  ATE.  By  concurrent  demonstration,  the  whole  is  greater 
than  the  sum  of  the  parts.  A  significant  set  of  factors  related  to  the  result  of  the 
integration  of  the  diagnostic  elements  must  be  evaluated  early. 


MAINTAINABILITY  DEMONSTRATIONS 
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MAINTAINABIUTY  DEMONSTRATIONS 


REQUIREMENT  #4.2 


CHECKLIST 


Poes  the  demonstration  plan  provide  a  100%  fault 
coverage  capability  across  all  levels  of  maintenance? 

®  Organizational  Level 
«  Intermediate  Level 
®  Depot  Level 

Are  the  failure  modes  to  be  demonstrated  and  criteria 
to  be  utilized  adequately  specified  for  each 
maintenance  level? 

Does  conducting  the  actual  demonstration  require  a 
level  of  manpower  training,  and/or  technical  informa¬ 
tion  above  and  beyond  that  which  will  be  provisioned? 

If  so,  why? 

E2f  Is  the  demonstation  structured  to  provide  an 
evaluation  of  the  diagnostic  capabilities  as 
an  integrated  system? 

E2f  Do  the  subject  plans  demonstrate  an  integrated, 

cohesive  maintenance  flow  in  terms  of  demonstrating 
how  a  fault  would  be  detected?  For  example,  at  the 
Organizational  Level  and  the  subject  repair  effected  at 
the  Depot  Level?  Is  a  systems  approach  to  the 
maintenance  process  taken  in  the  overall  approach  to 
demonstration  planning? 


REVIEWS 


REQUIREMENT  #5 


CONDUCTING  DESIGN  REVIEWS 


OVERVIEW 

During  the  acquisition  of  a  weapon  system  there 
are  at  least  eight  formal  technical  reviews  and 
audits,  which  may  be  conducted  by  the  contractor 
for  the  Government  Program  Manager.  As  in  the 
diagnostic  design  process,  there  is  a  tendency  to 
conduct  separate  reviews  and  audits  based  upon 
the  function  being  addressed.  This  particularly 
refers  to  logistics,  reliability,  maintainability, 
testability,  human  engineering,  and  safety. 
Integration  of  these  reviews  and  audits  to  address 
diagnostic  issues  is  a  must.  MIL-STD-1521  is  the 
prime  document  which  defines  the  issues  to  be 
addressed  at  each  of  these  formal  reviews.  At 
present,  these  checklists  are  inadequate  in  terms 
of  both  testability  and  diagnostics  and,'  thus,  these 
reviews  and  audits  may  not  serve  their  purpose. 
Additional  guidance  must  be  given  to  both  the 
government  and  the  contractor  in  order  to  alleviate 
this  problem. 

Informal  reviews  are  often  required.  Guidance  for 
these  informal  reviews  can  be  drawn  from  formal 
review  guidance. 


TESTABILITY/ 

DIAGNOSTICS 


I 


PROGRAMMATIC  | 

REQUIREMENTS  | 

DESIGN  I 

ASSESSMENT  | 

REVIEWS  1 

EVALUATION  | 

MATURATION  | 

IMPORTANT  CONSIDERATIONS  TO  BE  ADDRESSED 
Regmt. 

5.1  Technical  reviews  and  audits  niust  address  all  facets  which  affect  the 
performance  of  the  diagnostic  capability. 

Conduct  the  review  and  audit  of  testability  and  diagnostic  functions 
as  an  integral  part  of  system  engineering. 
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CONDUCTING  TECHNICAL  REVIEWS  AND  AUDITS 


REQUIREMENT  #5.1 


WEAPON 

SYSTEM 

ACQ.  PHASE 

OPER. 

REQMTS. 

CONCEPT 

EXPLOR. 

DEM/ 

VAL 

FSD 

PROD/ 

DEPL. 

WEAPON 

SYSTEM 

ACTIVITIES 

ZS  AAA  ZS 

SCP  DCP  PREL  DETAIL  lOT&E 

DESIGN  DESIGN 

DIAGNOSTIC 

ACTIVITIES 

A  AAAAAA  A 

SRR  SDR  PDR  CDR  TRR  PRR  FCA  PCA 

DIAGNOSTIC  ACTIVITY 

Technical  reviews  and  audits  are  an  important  factor  in  assuring  that  the 
government  is  furnished  with  a  weapon  system  which  meets  its  requirements. 

PROCEDURE 


MIL-STD-1521  lists  10  formal  technical  reviews  and  audits.  Of  these  10,  eight 
are  considered  critical  in  the  achievement  of  a  satisfactory  diagnostic  capability.  The 
following  guidance  supplements  and  expands  the  guidance  contained  in  MIL-STD-1521, 
Technical  Reviews  and  Audits  for  Systems,  Equipments,  and  Computer  Software. 

Although  design  reviews  are  recognized  as  being  important  to  verify  design 
before  production,  the  lack  of  depth  in  these  reviews  is  alarming.  The  cause  of  these 
inadequate  reviews  must  be  shared  by  both  the  contractors  and  the  government. 
Contractually,  the  government  rarely  requires  the  contractor  to  do  a  comprehensive 
technical  review,  and  the  contractor  does  not  do  so  unless  required,  even  though  it  may 
be  cost  effective  from  his  point  of  view.  Even  when  the  right  words  are  used,  the  end 
results  depend  largely  on  corporate  policy  to  allocate  sufficient  resources  to  perform  a 
detailed  analysis  of  the  design  and  associated  processes. 


CONDUCTING  TECHNICAL  REVIEWS  AND  AUDITS  REQUIREMENT  #5.1 

GUIDANCE 


Guidance  relating  to  these  various  reviews  is  contained  in  the  appendices  to 
MIL-STD-1521 .  Because  these  appendices  do  not  address  all  aspects  of  testability  and 
diagnostics,  some  supplemental  information  is  included  in  the  following  paragraphs. 

System  Requirements  Review  (SRR) 

The  objective  of  this  review  Is  to  ascertain  the  adequacy  of  the  contractor’s 
efforts  in  defining  system  requirements.  It  will  be  conducted  when  a  significant  portion  of 
the  system  functional  requirements  has  been  established. 

The  diagnostic  capability  review  portion  of  the  SRR  will  analyze  the  system 
items  that  are  related  to  diagnostics.  The  following  Items  will  be  reviewed,  as  appropriate: 

o  Mission  and  Requirements  Analysis 
o  Functional  Flow  Analysis 
o  Preliminary  Requirements  Allocation 
o  System/Cost  Effectiveness  Analysis 
o  Trade  Studies 
o  Synthesis 

o  Logistic  Support  Analysis 
0  Specialty  Discipline  Studies 
o  Generation  of  Specifications 
o  Program  Risk  Analysis 
o  Integrated  Test  Planning 
o  Technical  Performance  Measurement  Planning 
o  Engineering  Integration 
o  System  Safety 
o  Human  Factors  Analysis 
o  Life  Cycle  Cost  Analysis 
o  Manpower  Requirements/Personnel  Analysis 
o  Milestone  Schedules. 

The  diagnostic  capability  review  should  address  the  impact  of  the  results  of  the 
items  listed  above  on  the  diagnostic  pieces  listed  below. 

o  Designed-In  Reliability,  Prognostics,  and  Testability 
o  Self-Test,  Built-In  Test,  System  Integrated  Test 
o  Support  Equipment,  Maintenance  Aids 
0  Technical  Data 
o  Personnel  Skill  Requirements 
o  Training  and  Training  Devices. 
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System  Design  Review  (SDR) 

This  review  shall  be  conducted  to  e^'aluate  the  optimization,  correlation, 
completeness,  and  risks  associated  with  the  allocated  technical  requirements.  Also 
included  is  a  summary  review  of  the  system  engineering  process  which  produced  the 
allocated  technical  requirements  and  of  the  engineering  planning  for  the  next  phase  of 
effort.  Basic  manufacturing  considerations  will  be  reviewed,  and  planning  for  production 
engineering  in  subsequent  phases  will  be  addressed.  This  review  will  be  conducted  when 
the  system  definition  effort  has  proceeded  to  the  point  where  system  characteristics  are 
defined  and  the  Configuration  Items  (Cl)  are  identified. 

Specific  diagnostic  considerations  relate  to: 

o  Optimizing  the  diagnostic  capability  (changes  after  Dem/Val  usually  are  more 
costly  and  time  consuming) 

o  Preparation  of  a  Maturation  Plan 

o  Preparation  of  a  System  Specification  which  provides  a  capability  for 
addressing  necessary  FD/Ft  requirements  at  each  level  of  maintenance 

o  Allocation  of  diagnostic  requirements  for  each  diagnostic  element 

o  Review  of  the  software  requirements  specification  to  assure  that  embedded 
diagnostic  software  considerations  are  included. 

Preliminary  Design  Review  (PDR) 

This  review  shall  be  conducted  for  each  Configuration  Item  or  aggregate  of 
Configuration  Items  to:  (1)  evaluate  the  progress,  technical  adequacy,  and  risk  resolution 
(on  a  technical,  cost,  and  schedule  basis)  of  the  selected  design  approach;  (2)  determine 
its  compatibility  with  performance  and  engineering  specialty  requirements  of  the  Hardware 
Configuration  Item  (HWCI)  development  specification;  (3)  evaluate  the  degree  of  definition 
and  assess  the  technical  risk  associated  with  the  selected  manufacturing 
methods/processes;  and,  (4)  establish  the  existence  and  compatibility  of  the  physical  and 
functional  interfaces  among  the  Configuration  Item  and  other  items  of  equipment,  facilities, 
computer  software,  and  personnel.  For  Computer  Systems  Configuration  Items  (CSCIs), 
this  review  will  focus  on:  (1)  the  evaluation  of  the  progress,  consistency,  and  technical 
adequacy  of  the  selected  top-level  design  and  test  approach;  (2)  compatibility  between 
software  requirements  and  preliminary  design;  and,  (3)  on  the  preliminary  version  of  the 
operation  and  support  documents. 

In  addition,  the  following  items  in  the  diagnostic  area  should  be  presented  at  the 
appropriate  depth  for  review. 
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o  Preliminary  Failure  Modes  and  Effects  Analysis 

o  Design  data  analyses  for  BIT/SIT  Integrcited  diagnostics,  including 
requirements  and  preliminary  design  verification  results 

o  Maintenance  concept  for  the  portion  of  the  system  being  reviewed  and  its 
traceability  to  the  system  maintenance  concept 

o  Operational  maintenance  functions 

o  Results  of  the  analysis  of  the  inherent  (intrinsic)  testability  of  the  preliminary 
design 

o  Allocation  of  qualitative  and  quantitative  requirements 
o  Criteria  for  external  diagnostic  elements 
o  Trade-off  studies 

o  Cost/System  Effectiveness  Modeling  and  Life  Cycle  Cost  Analysis 

o  Preliminary  Logistic  Support  Analysis,  including  task  analysis  and  related 
personnel  and  support  equipment  information 

o  Impact  of  diagnostics  on  maintenance  man-hours  required 

o  Evaluation  of  alternatives 

o  Test  and  evaluation  plans. 

Critical  Design  Review  (CDR) 

This  review  shall  be  conducted  for  each  Configuration  Item  when  detail  design  is 
essentially  complete.  The  purpose  of  this  review  will  b©  to:  (1)  determine  that  the  detail 
design  of  the  Configuration  Item  under  review  satisfies  the  performance  and  engineering 
specialty  requirements  of  the  HWCI  development  specifications;  (2)  establish  the  detail 
design  compatibility  among  the  configuration  and  other  items  of  equipment,  facilities, 
computer  software  and  personnel;  (3)  assess  Configuration  Item  risk  areas  (on  a 
technical,  cost,  and  schedule  basis);  (4)  assess  the  results  of  the  producibility  analyses 
conducted  on  system  hardware;  and,  (5)  review  the  p'^eliminary  hardware  product 
specifications.  For  CSCIs,  this  review  will  focus  on  the  determination  of  the  acceptability 
of  the  detailed  design,  performance,  and  test  characteristics  of  the  design  solution,  and  on 
the  adequacy  of  the  operation  and  support  documents.  The  CDR  shall  be  conducted  on 
each  Configuration  Item  prior  to  fabrication/production/coding  release  to  ensure  that  the 
detail  design  solutions,  as  reflected  in  the  Draft  Hardware  Product  Specification,  Software 
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Detail  Design  Document  (SDDD),  Data  Base  Design  Document(s)  (DBDD),  interface 
Design  Document(s)  (IDD),  and  engineering  drawings,  satisfy  requirements  established  by 
the  Hardware  Development  Specification  and  Software  Top-Levei  Design  Document 
(STLDD).  The  CDR  shali  be  heid  after  the  Computer  Software  Operator’s  Manual 
(CSOM),  Software  User’s  Manual  (SUM),  Computer  System  Diagnostic  Manual  (CSDM), 
Software  Programmer’s  Manual  (SPM),  and  Firmware  Support  Manual  (FSM)  have  been 
updated  or  newly  released. 

It  is  desired  at  each  CDR  to  provide  as  much  assurance  as  practicable  that 
mission  critical  FD/FI  thresholds  are  realized  and  that  all  diagnostic  requirements  are 
satisfied:  e.g.,  that  1 00%  diagnostic  capability  wili  exist  for  each  Cl  in  the  system.  While  it 
probably  will  not  be  practicable  to  certify  that  this  will  exist,  the  following  data  should  be 
presented  as  an  extension  of  the  information  presented  at  the  PDR. 

o  Detailed  fault  detection/fault  isolation  analyses  that  identify  the  extent  to 
which  BIT/SIT  detect  and  isolate  faults  and  which  identify  those  failures  that 
will  require  support  equipment  and/or  manual  methods  to  detect  and/or 
isolate. 

o  Diagnostic  allocations  in  Part  11  Cl  specifications  to  the  LRU  and  SRU  level. 
Traceability  of  those  requirements  to  the  Part  I  Cl  System  Specification 
should  be  demonstrated.  Note:  Flexibility  to  reallocate  diagnostic 
allocations  until  product  baseline  is  established  at  PCA  should  be  provided 
within  the  envelope  of  system  requirements. 

o  Definition  of  the  maintenance  plan/concept  for  the  Cl,  together  with 
supporting  LSA  documentation,  including  support  requirement  and  level-of- 
repair  analysis  results.  Logistic  simulation  results  should  be  presented  to 
substantiate  the  plan/concept. 

o  Presentation  of  testability  analysis/assessment  results  for  the  Cl  design  to 
substantiate  the  fault  detection/  fault  isolation  analysis. 

o  Early  program  Failure  Identification,  Prevention,  and  Detection  (FIPAD) 
analyses  applicable  to  the  Cl  should  be  presented  to  assist  in  verifying 
diagnostic  capability. 

o  Review  detailed  Maintainability  Demonstration  Plan  for  inclusion  of 
diagnostic  capability  test  requirements 

o  Appropriate  updates  to  the  items  reviewed  during  the  PDR. 
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Test  Readiness  Review  (TRR) 

This  review  is  conducted  for  each  CSCI  to  determine  whether  the  software  test 
procedures  are  complete  and  to  assure  that  the  contractor  is  prepared  for  formal  CSCI 
testing.  Software  test  procedures  are  evaluated  for  compliance  with  software  test  plans 
and  descriptions  and  for  adequacy  in  accomplishing  test  requirements.  At  TRR  the 
contracting  agency  also  reviews  the  results  of  informal  software  testing  and  any  updates  to 
the  operation  and  support  documents.  A  successful  TRR  is  predicated  on  the  contracting 
agency’s  determination  that  the  software  test  procedures  and  informal  test  results  form  a 
satisfactory  basis  for  proceeding  into  formal  CSCI  testing. 

The  diagnostic  segment  of  the  system/CI  TRR(s)  shall  be  a  formal  review  of  the 
contractor’s  readiness  to  begin  formal  diagnostics-related  CSCI  testing.  It  is  conducted 
after  the  software  test  procedures  are  available  for  diagnostics-related  CSCI,  such  as  Cl 
BIT,  System  BIT,  SIT,  etc.,  and  after  computer  system  component  (CSC)  integration 
testing  is  complete. 

The  items  to  be  reviewed  include: 

1.  Requirement  Changes  - 

Any  changes  to  BIT,  SIT,  or  testability  requirements  contained  in  the 
system/CI  Software  Requirement  Specification  or  Interface  Requirements 
Specification  that  have  not  been  approved  and  which  impact  CSCI  testing. 

2.  Design  Changes  -- 

Any  changes  made  to  the  BIT,  SIT,  or  testability  design  parameters 
contained  in  the  Software  Top-Level  Design  Document  (STLDD),  Software 
Detail  Design  Document  (SDDD),  Interface  Design  Document(s)  (IDD)  since 
the  PDR  and  CDR  which  impact  CSCI  testing. 

3.  Software  Test  Plans  and  Descriptions  - 

Any  changes  to  the  embedded  diagnostic  element  portion  of  the  approved 
Software  Test  Plans  (STP)  and  Software  Test  Descriptions  (STD). 

4.  Software  Test  Procedures  -- 

Test  procedures  to  be  used  in  conducting  BIT  and/or  SIT  test  effectiveness 
validation  as  part  of  the  CSCI  testing,  including  retest  procedures  for  test 
anomalies  and  corrections. 

5.  Integration  Test  Cases,  Procedures,  and  Results  -C  .  - 
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Any  embedded  diagnostic  element  CSC  (e.  g.,  BIT  components,  SIT 
components)  integration  test  cases,  and  procedures  used  In  conducting 
informal  diagnostic  element  CSC  integration  tests  and  the  test  results. 


6.  Software  Test  Resources  ~ 

Status  of  any  software  test  resources  that  are  required  specifically  for 
embedded  diagnostic  element  CSCI  testing.  Such  resources  may  include 
diagnostic  test  personnel  and  supporting  test  software  and  materials, 
including  software  test  tool  qualification  and  review  of  the  traceability 
between  requirements  and  their  associated  tests. 

7.  Test  Limitation  -- 

Identification  of  all  software  test  limitations  associated  with  embedded 
diagnostic  element  CSCI/CSC  testing. 

8.  Software  Problems  -- 

Summary  of  embedded  diagnostic  element  software  problem  status, 
including  all  known  discrepancies  of  the  CSCI  and  test  support  software. 

9.  Schedules  - 

Schedules  for  the  remaining  embedded  diagnostic  element  software 
milestones. 

Production  Readiness  Review  (PRR) 

This  review  is  intended  to  determine  the  status  of  completion  of  the  specific 
actions  which  must  be  satisfactorily  accomplished  prior  to  executing  a  production  go- 
ahead  decision.  The  review  is  accomplished  in  an  incremental  fashion  during  the  Full- 
Scale  Development  Phase-usually  two  initial  reviews  and  one  final  review,  to  assess  the 
risk  in  exercising  the  production  go-ahead  decision.  In  its  earlier  stages,  the  PRR 
concerns  Itself  with  gross-level  manufacturing  concerns,  such  as  the  need  for  identifying 
high-risk/low-yield  manufacturing  processes  or  materials  or  the  requirement  for 
manufacturing  development  effort  to  satisfy  design  requirements.  The  reviews  become 
more  refined  as  the  design  matures,  dealing  with  such  concerns  as  production  planning, 
facilities  allocation,  incorporation  of  producibility-oriented  changes,  identification  and 
fabrication  of  tools/test  equipment,  long-lead  item  acquisition,  etc.  Timing  of  the 
Incremental  PRRs  is  a  function  of  program  posture  and  is  not  specifically  locked  into  other 
reviews.  The  diagnostic  consideration  concerns  the  use  of  any  of  the  external  diagnostic 
elements  (e.g.,  ATE)  in  the  production  testing  environment. 
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Functional  Configuration  Audit  (FCA) 

This  is  a  formal  audit  to  validate  that  the  development  of  a  Configuration  Item 
has  been  completed  satisfactorily  and  that  the  Configuration  Item  has  achieved  the 
performance  and  functional  characteristics  specified  in  the  functional  or  allocated 
configuration  identification.  In  addition,  the  completed  operation  and  support  documents 
shall  be  reviewed. 

The  FCA  is  normally  conducted  on  a  prototype  or  preproduction  item.  The  FCA 
validates  that  the  item  meets  its  specified  performance  requirements  and  is  ready  for 
production  and  acceptance  into  Air  Force  inventory,  it  is  imperative  that  the  diagnostic 
capability  be  validated  against  its  specified  performance  requirements,  so  that  any 
diagnostic  capability  deficiencies  can  be  identified  and  corrected  before  the  item  proceeds 
into  production  and  is  then  deployed. 

Physical  Configuration  Audit  (PCA) 

This  is  a  technical  examination  of  a  designated  Configuration  Item  to  verify  that 
the  Configuration  Item  "as  built"  conforms  to  the  technical  documentation  which  defines 
the  Configuration  Item. 

After  successful  completion  of  the  audit,  all  subsequent  changes  to  the 
diagnostic  eiements  are  processed  by  an  engineering  change  action.  The  PCA  also 
determines  that  the  diagnostic  element  acceptance  testing  prescribed  by  the 
documentation  is  adequate  for  acceptance  of  the  production  units  by  quality  assurance 
activities.  The  procedures  for  conducting  a  PCA  are  contained  in  MIL'STD-1521, 
Appendix  H.  Sample  PCA  Certification  Attachment  Checklists  are  contained  in  MIL-STD- 
1521,  Appendix  I. 
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CHECKLIST 

of  Does  the  contractor  have  a  corporate  policy 
identifying  the  procedures  for  internal  reviews, 
as  well  as  customer  required  reviews? 

O'  Is  emphasis  being  placed  on  technical  inter¬ 
change  meetings  between  contractor  and  customer 
rather  than  large-scale  reviews? 

□f  Are  qualified  diagnostic  technical  experts,  who 
can  challenge  the  design  and  assess  risks, 
included  in  these  reviews? 

O'  Are  the  diagnostic  reviews  held  as  an  integral 
part  of  the  prime  system  review,  but  in 
a  timely  manner  that  allows  change  (if  necessary) 
in  the  diagnostic  equipment  or  process? 


EVALUATION  REQUIREMENT  §6 


CONDUCTING  TEST  AND  EVALUATION 
OVERVIEW 


During  the  development  of  a  weapon  system,  a 
number  of  tests  and  evaluations  are  conducted  by 
subcontractors,  the  prime  contractor,  and  the 
government.  Many  of  these  tests  address  the 
performance  of  the  diagnostic  capability.  It  is  not 
uncommon  that  these  tests  are  conducted 
separately  and,  thus,  do  not  address  the  entire 
diagnostic  capability.  Oftentimes  the  entire 
diagnostic  capability  is  not  delivered  in  time  to  test 
and  evaluate  the  diagnostic  capability  as  a  whole. 
During  the  major  tests  and  evaluations  (e.g., 
DT&E,  OT&E)  as  much  as  possible  of  the  entire 
diagnostic  capability  should  be  included. 
Integrated  demonstration,  test,  and  evaluation  is 
required. 


Coordination  of  all  test  and  evaluations, 
including  demonstrations,  can  be  accomplished 
through  the  preparation  of  an  Integrated  Test  Plan. 


IMPORTANT  CONSIDERATIONS  TO  BE  ADDRESSED 
Reqmt. 

6.1  Prepare  an  Integrated  Test  Plan,  which  Includes  the  requirements 


j  MATURATl" 


TESTABILITY/ 

DIAGNOSTICS 

"n — 


PROGRAMMATIC 
'■  'I  REQUIREMENTS 


DESIGN 


I  ASSESSMENT 


h-c  REVIEWS 


for  a  Test  and  Evaluation  Master  Plan. 

I 


6.2/6.3  Assure  that  formal  test  and  evaluations  address  the  entire 
diagnostic  capability. 


PREPARATION  OF  THE  TEMP 
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The  requirements  for  diagnostics  test  and  evaluation  are  identified,  scheduled 
and  integrated  into  the  Test  and  Evaluation  Master  Plan  (TEMP)  by  the  contractor. 

PROCEDURE 


The  Government  Program  Manager  should  ensure  that  adequate  requirements 
are  in  place  for  diagnostics  test  and  evaluation.  The  TEMP  is  a  living  document  -  its 
preparation  goes  through  many  iterations  as  the  program  proceeds  through  Concept 
Exploration  Phase  studies,  Demonstration  and  Validation,  Full-Scale  Development, 
Production  and  Deployment.  With  each  Iteration,  plans  for  diagnostic  T&E  should  become 
firmer,  better  defined,  and  with  target  milestone  dates  attached. 

Because  test  and  evaluation  is  a  major  cost  and  schedule  driver,  adequate 
planning  is  essential  long  before  its  start.  Test  planning  between  subcontractors,  the 
prime  contractor,  and  the  government  should  start  with  program  initiation.  To  ensure  a 
successful  integrated  test  program,  close  coordination  is  required  between  the 
government,  the  prime  contractor,  and  all  subcontractors. 

GUIDANCE 

DoD  Directive  5000.3  requires  the  preparation  of  a  TEMP.  The  TEMP  is  a 
broad  plan  relating  test  objectives  to  required  system  characteristics  and  critical  issues, 
and  is  a  top-level  document  used  at  major  milestone  reviews  to  assess  the  adequacy  of 
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planned  test  and  evaluation.  The  TEMP  normally  covers  only  government-required  tests, 
and  does  not  provide  a  sufficient  level  of  detail  to  Identify  contractor  and  subcontractor 
tests.  In  an  attempt  to  control  the  test  program  at  the  contractor  and  subcontractor  level, 
contracts  may  contain  requirements  for  the  submittal  of  Individual  test  plans  for 
government  approval,  if  an  Integrated  Test  Plan  is  not  required,  these  individual  test  plans 
may  not  be  reviewed  for  duplicate  or  missing  test  activities,  resulting  in  an  inefficient  and 
costly  test  program. 

The  prime  contractor  should  be  responsible  for  the  preparation  and  updating  of 
an  Integrated  Test  Plan  (ITP).  To  develop  an  efficient  and  well  coordinated  Integrated 
Test  Plan,  the  prime  contractor  and  ail  subcontractors  should  jointly  participate  in  its 
preparation.  The  ITP  should  include  all  developmental  tests  to  be  performed  by  the  prime 
contractor  and  all  subcontractors  at  both  the  system  and  subsystem  levels.  The  ITP 
should  be  a  detailed  working-level  document  which  will  aid  in  identifying  risk,  duplication  or 
missing  test  activities,  and  provide  for  the  most  efficient  use  of  test  facilities  and  test 
resources.  In  developing  the  ITP,  the  purpose  and  time  phasing  of  each  individual  test 
should  be  carefully  examined.  Unnecessary  tests  should  be  eliminated  and  test 
schedules  should  be  adjusted  to  provide  sufficient  time  for  retest,  should  failures  occur. 
The  proper  sequencing  of  tests  is  necessary  to  ensure  completion  of  required  lower-level 
subcontractor  tests  prior  to  the  start  of  prime  contractor  tests.  Detailed  requirements  for 
diagnostics  T&E  should  be  Included  in  the  Integrated  Test  Plan.  These  requirements 
should  be  phased  T&E  of  all  diagnostic  elements  and  the  integration  of  the  elements.  The 
ITP  should  also  include  close  coordination  between  ail  T&E  activities. 

During  Development  Test  and  Evaluation  (DT&E)  the  contractor  and  the 
government  normally  conduct  separate,  dedicated  tests.  In  many  instances  these 
separate  test  periods  result  in  redundant  testing,  testing  which  is  not  user-oriented,  lack  of 
continuity  in  the  contractor’s  development  program,  and  a  lack  of  cooperation  between 
contractor  and  government  personnel.  In  order  to  increase  the  efficiency  of  DT&E,  the 
government  should  participate  in  some  of  the  contractor’s  testing.  This  will  help  eliminate 
redundant  testing,  reduce  the  length  of  DT&E  phases,  provide  more  user-oriented  test 
results,  and  result  in  a  more  mature  system  for  Operational  Tost  and  Evaluation  (OT&E). 

Most  test  schedules  are  planned  to  support  the  major  milestone  reviews  that 
occur  during  the  development  of  a  weapon  system.  The  tests  are  planned  to  provide 
positive  (successful)  test  results  for  presentation  at  these  milestone  reviews,  in  order  to 
obtain  approval  for  the  project  to  proceed  to  the  next  milestone.  This  leads  to  a  test 
philosophy  in  which  passing  tests  is  the  main  objective  of  the  test  program,  rather  than 
considering  the  engineering  need  for  the  test  or  the  technical  information  provided  by  the 
test  results.  As  a  result  of  this  philosophy,  test  schedules  tend  to  be  success-oriented, 
many  times  resulting  in  schedule  slippage  due  to  the  need  for  retest  or  a  lack  of  test 
assets. 
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Test  schedules  should  be  properly  phased  primanly  on  the  basis  of  engineering 
considerations,  rather  than  strictly  milestone-oriented.  The  purpose  or  objective  of  each 
test  should  be  considered  as  well  as  the  interrelation  of  various  tests  with  each  other.  As 
test  programs  progress,  many  tests  will  disclose  a  need  for  redesign  and  retest,  such  as 
redesign  of  System  Integrated  Test  (SIT)  or  Built-in  Test  (BIT)  hardware  or  software.  In 
some  instances  only  a  minor  correction  and  verification  test  will  be  required.  In  other 
cases  the  corrective  actions  may  be  extensive  and  require  significant  retest.  If  test 
schedules  have  not  allowed  sufficient  time  for  redesign  and  retest,  changes  and  retesting 
may  be  delayed  until  production  equipment  is  available.  If  the  changes  prove  incorrect 
and  additional  redesign  is  required,  production  units  may  have  to  be  retrofitted  and  a  large 
number  of  Engineering  Change  Proposals  (ECPs)  may  be  required  during  the  early 
phases  of  the  production  program.  Also,  due  to  the  sequential  nature  of  some  tests,  the 
performance  of  certain  tests  may  be  delayed  until  production,  possibly  resulting  in 
additional  ECPs. 

Since  the  start  of  certain  tests  may  be  dependent  upon  the  completion  of  others, 
critical  tests  should  be  identified  and  provisions  made  for  schedule  slippage  due  to  needed 
redesign  and  retest.  In  certain  cases  critical  test  schedules  can  be  accelerated  by 
pro  'iding  more  test  assets  or  additional  tost  facilities.  This  strategy  can  provide  significant 
leverage  to  reduce  the  overall  development  test  schedule.  Milestone  reviews  can  then  be 
planned  on  the  basis  of  realistic  test  schedules.  More  engineering-oriented  test  results 
showing  design  strengths  and  weaknesses  should  be  presented  at  design  reviews.  The 
review  should  discuss  design  weaknesses  and  how  they  have  been  or  will  be  corrected. 
The  overall  success  of  a  carefully  integrated  test  program  will  result  in  a  minimum  of 
resources  applied  to  testing  and  the  elimination  of  a  costly  ECP  or  retrofit  program  during 
production. 

In  accordance  with  OoO  Directive  5000.3,  test  and  evaluation  must  begin  as 
early  as  possible  in  the  system  acr uisition  process  to  reduce  acquisition  risks  and  to 
estimate  the  capability  of  the  system  under  development  to  meet  all  technical  and 
operational  requirements.  Critical  test  and  evaluation  issues  (to  include  all  effectiveness 
and  suitability  issues),  objectives,  methodologies  and  evaluation  criteria  are  defined  during 
the  initial  establishment  of  an  acquisition  program.  These  criteria  serve  to  define  the 
testing  required  for  each  phase  of  the  acquisition  process  and  provide  the  structure  to 
guide  the  testing  program.  Diagnostics  should  be  established  as  a  critical  suitability  and 
logistic  supportability  issue  in  order  to  provide  the  proper  degree  of  focus  on  diagnostics  T 
&  E.  For  example,  if  two-level  maintenance  is  a  system  requirement,  then  diagnostics 
capability  is  very  critical  to  achieving  that  requirement. 

Testing  must  be  planned  and  conducted  to  provide  quantitative  data  and  to 
minimize  the  need  for  subjective  interpretation  of  system  performance  and  suitability 
factors.  This  requires  early  planning  to  determine  the  number  of  test  articles  needed  as 
well  as  other  support  resources.  The  developer  and  the  test  and  evaluation  agencies 
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should  share  information  and  data  during  the  acquisition  process  to  establish  a  data  base 
allowing  progressive  evaluation  of  the  system. 

The  use  of  properly  validated  analysis,  models  and  simulation  is  strongly 
encouraged,  especially  during  development  phases  to  assess  the  areas  which  cannot  be 
observed  through  testing.  Use  of  these  methods  can  provide  early  projections  and  can 
reduce  testing  costs  by  supplementing  actual  tost  data. 

Developmental  Test  and  Evaluation  (DT&E)  is  the  T&E  conducted  throughout 
various  phases  of  the  acquisition  process.  This  will  ensure  the  acquisition  and  fielding  of 
an  effective  and  supportable  system  by  assisting  in  the  engineering  design  and 
development  and  verifying  attainment  of  technical  performance  specifications,  objectives 
and  supportability. 

Developmental  Test  and  Evaluation  also  includes  T&E  of  components, 
subsystems,  preplanned  product  improvement  (P^l)  changes,  hardware-software 
integration  and  related  software,  as  well  as  qualification  and  production  acceptance 
testing.  Test  and  evaluation  of  compatibility  and  interoperability  with  existing  or  planned 
equipment  or  systems  is  emphasized.  This  T&E  encompasses  the  use  of  models, 
simulations  and  testbeds,  as  well  as  prototypes  of  Full-Scale  Development  models  of  the 
system.  The  diagnostic  capability  associated  with  component,  assembly  and  subsystem 
DT&E  should  be  included  in  these  T&E  activities. 

Qualification  Testing  is  that  part  of  DT&E  which  verifies  the  design  and  the 
manufacturing  process  and  provides  a  baseline  for  subsequent  acceptance  tests.  This 
accomplishes  two  separate  functions: 

(1)  Preproduction  Qualification  Tests  are  formal  contractual  tests  that  ensure 
design  integrity  over  the  specified  operational  and  environmental  range.  These  tests 
usually  use  prototype  or  preproduction  hardware  fabricated  to  the  proposed  production 
design  specifications  and  drawings.  Such  tests  include  contractual  reliability  and 
maintainability  demonstration  tests  required  prior  to  production  release.  At  a  minimum, 
embedded  diagnostics  capabilities  and  the  interfaces  to  external  diagnostic  elements 
should  be  tested  and  evaluated  during  preproduction  qualification  tests.  As  a  goal,  the 
capability  of  external  diagnostic  elements  should  also  be  tested  and  evaluated. 

(2)  Production  Qualification  Tests  ensure  the  effectiveness  of  the  manufacturing 
process,  equipment  and  procedures.  These  tests  are  conducted  on  a  sample  lot  taken  at 
random  from  the  first  production  lot,  and  are  repeated  as  the  process  or  design  is  changed 
significantly,  and  when  a  second  or  alternate  source  is  brought  on  line.  These  tests  are 
also  conducted  against  contractual  requirements.  The  utilization  of  diagnostic  resources 
in  the  manufacturing  process  and  the  requirement  for  capture  of  diagnostic  data  from  the 
manufacturing  process  should  be  evaluated  during  production  qualification  testing. 
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The  completion  of  Preproduction  Qualification  Test  and  Evaluation  before 
Milestone  III  decisions  are  made  is  essential  and  will  be  a  critical  factor  in  assessing  the 
system’s  readiness  for  production. 

Operational  Test  and  Evaluation  (OT&E)  is  the  field  test,  under  realistic 
conditions,  of  any  item  (or  key  component)  of  weapons,  equipment  or  munitions  for  the 
purpose  of  determining  the  effectiveness  and  suitability  for  use  in  combat  by  typical 
military  users;  and  the  evaluation  of  the  results  of  such  tests.  Operational  testing  is 
accomplished  in  an  environment  as  operationally  realistic  as  possible.  The  entire 
diagnostic  capability  should  be  assessed  during  OT&E  as  well  as  the  integration  of  the 
diagnostic  capability. 

The  Test  and  Evaluation  Master  Plan  (TEMP)  must  clearly  specify  development 
and  operational  test  events.  I^owever,  DT&E  and  OT&E  are  not  necessarily  serial  phases 
in  the  evolution  of  a  weapon  system.  During  critical  acquisition  cycle  transitions,  elements 
of  DT&E  and  OT&E  may  be  combined  or  occur  in  parallel,  but  not  at  the  expense  of  either 
development  or  operational  test  realism  nor  before  sufficient  DT&E  can  reasonably  assure 
that  the  system  is  ready  to  enter  dedicated  operational  testing.  DT&E  may  continue  into 
the  Production  and  Deployment  Phase,  along  with  OT&E,  to  address  system 
enhancements,  correction  of  deficiencies,  or  modifications.  In  all  cases,  test  planning  for 
all  tost  phases  must  be  addressed  in  the  system  TEMP. 

Test  and  evaluation  planning  is  initiated  at  the  inception  of  the  development 
process  to  ensure  adequate  planning,  programming  and  budgeting  of  test  resources  and 
to  facilitate  test  scheduling  to  support  major  program  decision  milestones.  Reliability 
assurance  should  be  well  underway  before  the  initiation  of  system  performance  tests. 
System  deficiencies  must  be  addressed  through  a  dynamic,  well-documented,  and  tightly 
managed  test-analyze-fix  and  retest  program.  The  evaluation  of  embedded  diagnostic 
elements  should  be  injected  into  these  reliability  assurance  tests. 

A  TEMP  is  required  for  all  major  defense  acquisition  programs.  The  TEMP 
defines  and  integrates  test  objectives,  critical  issues,  systems  characteristics, 
responsibilities,  resources  and  schedules  for  test  and  evaluation.  Test  resource 
requirements  must  be  addressed  in  the  TEMP,  along  with  a  critical  analysis  of  any 
shortfalls  that  will  impede  the  full  test  and  evaluation  of  the  system.  The  need  for  and  the 
availability  of  the  various  diagnostic  elements  which  make  up  the  diagnostic  capability  is 
addressed  in  the  TEMP.  Plans  to  correct  existing  or  anticipated  test  resource  limitations 
are  also  included,  as  is  a  listing  of  evaluation  criteria  delineating  critical  parameters 
permitting  continuous  oversight  and  independent  assessment. 

DoD  5000.3-M-1  contains  the  guidelines  for  the  preparation  of  the  TEMP. 
Detailed  guidance  on  the  diagnostic  inputs  to  the  TEMP  are  provided  below. 


PREPARATION  OF  THE  TEMP  REQUIREMENT  #6.1 

Detailed  Guidance  •  Diagnostic  Inputs  to  TEMP 
Part  I  -  System  Details 

Section  2b.  Interfaces 

Establish  diagnostics  as  a  system  that  is  required  to  accomplish  the  mission. 

Section  3  -  Required  Technical  Characteristics 

Include  diagnostic  performance  leveis  in  the  listing  of  key  technical 
characteristics,  performance  goals  and  thresholds. 

Section  4  -  Required  Operational  Characteristics 

Include  diagnostics  performance  parameters  as  key  operational  effectiveness  and 
suitability  characteristics,  goals  and  thresholds. 

Part  II  -  Program  Summary 

Section  1  -  Management 

Identify  organizational  elements  responsible  for  diagnostics  T&E. 

Section  2  -  Integrated  Schedule 

Ensure  Diagnostics  T&E  are  included  and  conform  to  System  T&E  schedule. 

Part  III  -  DT&E 

Define  in  detail  the  diagnostic-related  test  objectives  for  DT&E.  Relate  those 
objectives  to  system  operational  concept. 

Section  1  -  Critical  Technical  Characteristics 

Discuss  the  availability  of  diagnostic  elements  as  it  impacts  the  ability  to  evaluate 
the  total  diagnostic  capability.  Determine  the  criticality  of  the  availability  of  the 
diagnostic  elements.  Describe  diagnostic  workarounds  and  the  risks  (associated 
with  those  workarounds)  being  taken  by  not  being  able  to  evaluate  the  diagnostic 
capability  provided  by  the  integration  of  all  of  the  diagnostic  eiements. 

Section  2  -  DT&E  to  Date 

Summarize  diagnostics-related  DT&E  already  conducted. 

Describe  test  articles,  with  emphasis  on  how  they  differ  from  planned  production 
articles 
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Section  4  -  Future  DT&E 

Discuss  ail  remaining  diagnostics  DT&E  planned,  including  description  of  system 
diagnostics  and  diagnostics  resources  to  be  utilized  in  DT&E.  Identify  how  these 
differ  from  the  production  unit/system  and  diagnostic  resources.  Define  detailed 
diagnostics  DT&E  objectives,  DT&E  events,  scope  of  testing  and  basic  scenarios. 
Relate  test  objectives  to  test  procedures. 

Part  iV  -  OT&E 

Discuss  all  planned  diagnostics-related  OT&E  activities  and  their  objectives. 
Include  planning  from  lOT&E  through  the  FOT&E  during  initial  production  and 
deployment  which  address: 

o  The  ability  of  the  diagnostic  capability  to  support  operational  effectiveness  and 
suitability 

o  Identification  of  diagnostic  deficiencies  in  the  production  system  (including 
deficiencies  in  all  diagnostic  elements). 

Include  diagnostic  OT&E  considerations  in  the  following  sections. 

Section  1  -Critical  Operational  Issues 
Section  2  -OT&E  to  Date 
Section  3  -  Future  OT&E 

b.  OT&E  Objectives 

c.  OT&E  Events/Scope  of  Testing/Basic  Scenarios 

Part  V  -  Test  and  Evaluation  Resource  Summary 

Identify  the  key  resources  for  diagnostic  capability  DT&E,  OT&E  and  Produf'fion 
Acceptance  Test  and  Evaluation  (PAT&E)  that  are  unique  to  the  program. 

Section  1  -  Test  Articles 

Analyze  the  system  TEMP  to  identify  the  actual  number  of  test  articles  planned 
for  DT&E,  OT&E  and  PAT&E  and  the  type  of  article  planned  (prototype,  pilot 
production,  production  articles).  Determine  if  this  is  sufficient  for  diagnostics- 
related  DT&E,  OT&E  and  PAT&E. 

Section  2  -  Test  Support  Equipment 

Briefiy  describe  the  special  support  resources  (instrumentation,  test  sites, 
facilities,  military  maintenance  manpower)  required  for  diagnostics  T&E,  and 
briefly  describe  the  steps  being  taken  to  acquire  them. 
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CHECKLIST 

Have  diagnostics— related  inputs  to  the  TEMP 
been  prepared? 


Is  planning  for  diagnostics  T<feE  consistent 
with  system— level  planning? 

E3f  Have  all  diagnostics— related  T&E  risks, 
critical  items  and  special  resource 
requirements  been  adequately  resolved? 

O'  Does  planned  diagnostic  element  T&E  relate 
closely  to  the  actual  diagnostic  elements  to  be 
deployed  with  the  system? 

Is  there  a  logical  relationship  between  diagnostic 
T&E  activities  and  the  Diagnostic  Maturation  Program 
(i.e.,  are  diagnostic  T<ScE  results  going  to  be  used  as  a 
basis  for  the  maturation  of  the  diagnostic  capability)? 

O'  Have  all  T<ScE  activities  been  analyzed  to  assess 

the  feasibility  of  evaluating  the  diagnostic  capability 
of  the  system  (e.g.,  can  Reliability  Growth  Testing 
contribute  any  diagnostic  related  T&E  information)? 


DEVELOPMENT  TEST  ic  EVALUATION 


REQUIREMENT  #6.2 


WEAPON 
SYSTEM 
ACQ.  PHASE 


WEAPON 

SYSTEM 

ACTIVITIES 


DIAGNOSTIC 

ACTIVITIES 


OPER. 

CONCEPT 

DEM/ 

REQMTS. 

EXPLOR. 

VAL 

PROD/ 

DEPL 


DIAGNOSTICS 

DT&E 


DIAGNOSTIC  ACTIVITY 

Evaluate  diagnostics  performance  characteristics  during  Development  Test  and 
Evaluation  (DT&E)  activities  in  order  to  determine  diagnostic  capabilities  achieved  and  to 
identify  deficiencies  in  the  diagnostic  capability.  Diagnostics  DT&E  should  also  attend  to 
the  capability  achieved  by  the  integration  of  the  various  planned  diagnostic  elements 
(performance  monitors,  BIT/SIT,  testing:  automatic  and  manual,  maintenance  aids, 
technical  information  and  training  (skills))  into  a  comprehensive,  cohesive,  diagnostics 
subsystem. 

PROCEDURE 

Development  Test  and  Evaluation  is  the  T&E  conducted  by  the  contractor 
throughout  various  phases  of  the  acquisition  process  to  ensure  the  acquisition  and  fielding 
of  an  effective  and  supportable  system  by  assisting  in  the  engineering  design  and 
development  and  verifying  attainment  of  technical  performance  specifications,  objectives 
and  supportability. 

Development  Test  and  Evaluation  also  includes  T&E  of  components, 
subsystems  and  preplanned  product  improvement  (P^l)  changes,  hardware-software 
integration  and  related  software,  as  well  as  qualification  and  production  acceptance 
testing.  Test  and  evaluation  of  compatibility  and  interoperability  with  existing  or  planned 
equipment  and  systems  is  emphasized.  Development  Test  and  Evaluation  encompasses 
the  use  of  models,  simulations,  and  testbeds,  as  well  as  prototypes  or  Full-Scale 
Development  models  of  the  system. 
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GUIDANCE 


The  thrust  of  the  Integrated  Diagnostics  Process  with  respect  to  DT&E  is  to 
include/inject  diagnostic  performance  evaluation  into  the  mainstream  of  DT&E  activities. 
This  is  done  such  that  diagnostic  performance  can  be  evaluated,  deficiencies  pinpointed, 
and  corrective  action  implemented  while  the  system  is  still  in  development. 

The  diagnostics  DT&E  effort  assists  the  diagnostic  design  and  development 
process,  and  verifies  attainment  of  diagnostic  technical  performance  specifications, 
requirements,  and  objectives.  As  such,  it  is  an  integral  part  of  the  weapon  system  design 
process.  Through  the  provision  of  diagnostics  DT&E  data,  there  is  a  feedback  reiterative 
loop  back  into  the  integrated  diagnostics  activities  in  process,  including  the  diagnostic 
system  engineering  analysis;  diagnostic  risk  analysis,  allocation  of  diagnostic  goals; 
diagnostic  trades  for  system  optimization;  diagnostic  design  trades;  and  the  identification 
and  performance  of  diagnostic  design  tasks.  Through  this  methodology,  the  diagnostic 
design  is  corrected,  improved,  or  updated,  and  the  diagnostic  design  matures. 

Sufficient  diagnostics  DT&E  must  be  accomplished  before  the  Milestone  III 
decision  to  proceed  to  production.  This  will  ensure  that  the  major  specified  diagnostics 
design  and  development  requirements  for  the  Full-Scale  Development  Phase  have  been 
met,  with  respect  to  performance  requirements  and  specifications  contained  in  program 
documents. 

The  involvement  of  the  Government  Program  Manager  includes  establishing  the 
requirement  for  diagnostic  parameter  evaluation  during  DT&E,  monitoring  of,  and/or 
participation  in,  the  diagnostic  DT&E  activities  and  evaluation  of  DT&E  data  and  results. 
(In  order  to  establish  the  requirement  for  diagnostics  DT&E,  the  contract  Statement  Of 
Work  and  CDRL  must  include  appropriate  requirement  statements.)  The  contractors 
Integrated  Test  Plan  and  Diagnostics  Maturation  Plan  should  not  be  approved  by  the 
Government  Program  Manager  until  adequate  detail  Is  provided  for  evaluation  j.’  the 
specific  T&E  procedures,  data  bases,  models  and  test  articles,  scope  of  testing,  etc.  The 
Government  Program  Manager  should  ensure  that  diagnostic  inputs  to  the  TEMP  have 
been  accomplished  and  that  those  inputs  are  consistent  with  the  Diagnostics  Maturation 
Plan. 


The  Government  Program  Manager  should  bo  as  actively  involved  in 
diagnostics  DT&E  as  is  necessary  to  ensure  that  valid  tests  are  being  performed,  valid 
results  documented,  and  valid  data  accumulated  and  to  ensure  that  a  closed-loop  analytic 
approach  is  used  to  pinpoint  and  correct  diagnostic  deficiencies.  The  Government 
Program  Manager  should  also  ensure  that  every  opportunity  Is  being  taken  to  evaluate 
diagnostics  related  parameters.  This  may  involve  a  wide  range  of  test  activities,  including 
reliability  tests,  performance  tests,  human  factor  tests,  etc.  Basically  whenever  a  system, 
subsystem  or  component  is  being  operated,  it  is  subject  to  a  failure.  The  diagnostics 
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requirements  associated  with  dealing  with  the  failure  should  be  viewed  as  an  opportunity 
to  assess  the  diagnostic  capability. 

The  Government  Program  Manager  should  evaluate  the  results  and  data  from 
DT&E.  Based  upon  the  results  and  data,  critical  areas  should  be  identified.  Appropriate 
modifications  should  be  made  to  the  TEMP  with  respect  to  planning  for  OT&E  activities. 
Any  significant  deviations  from  the  Diagnostic  Maturation  Plan  and  attainment  of  specified 
diagnostic  capability  levels  should  be  tracked. 

The  Government  Program  Manager  must  determine,  based  upon  the  scope, 
basis  and  results  of  the  T&E  activity,  what  the  degree  of  confidence  is  that  the  deployed 
diagnostic  capability  will  achieve  operational  suitability  and  logistic  supportability 
requirements  of  the  system.  If  either  the  scope  of  testing,  the  basis  of  testing,  or  the 
results  of  testing  are  far  from  the  deployed  capability,  then  confidence  should  be  low,  and 
diagnostics  should  be  identified  as  a  risk  item  in  the  TEMP.  Specific  efforts  should  be 
taken  to  resolve  or  reduce  this  risk.  Scope  of  testing  here  refers  to  the  evaluation  of  the 
diagnostic  capability  achieved  as  a  result  of  the  integration  of  diagnostic  elements  across 
system  assembly  levels  and  maintenance  levels.  Therefore,  the  full  scope  of  diagnostics 
T&E  is: 


The  scope  of  diagnostic  T&E  should  include  fault  detection  and  isolation 
accuracy  and  timeliness  provided  by  performance  monitoring,  BIT/SIT,  automatic  and 
manual  testing,  technical  information  and  maintenance  aids,  maintenance  procedures, 
manpower,  personnel  and  skill  levels  at  the  system,  subsystem,  LRU/LRM,  SRU  levels 
across  planned  maintenance  echelons  (Organizational,  Intermediate  and  Depot). 

Any  deviation  from  this  full  scope  of  T&E  means  that  full  confidence  cannot  be 
ascribed  to  the  planned  diagnostic  capability. 

These  factors  must  be  evaluated  in  terms  of  their  interrelationships  also.  For 
example,  the  Depot-level  test  capability  may  be  deemed  a  low-risk,  non-essential  portion 
of  the  diagnostics  capability.  Suppose  then  that  the  false  alarm  rate  proves  to  be  high  and 
the  ambiguity  resolution  proves  to  be  poor.  Soon,  the  Depot  capability  becomes  critical, 
with  large  numbers  of  UUTs  awaiting  test,  and  spares  supplies  being  depleted. 

"Basis  of  testing"  refers  to  the  extensiveness  of  test  and  the  procedures  utilized 
for  T&E.  The  basic  issue  is  whether  the  T&E  was  performed  such  that  confidence  can  be 
ascribed  to  the  results  of  the  test.  It  is  usually  unrealistic  to  plan  for  exhaustive  testing  of 
diagnostics  because  of  the  many  and  varied  failure  modes  to  which  the  system  is 
subjected.  Fault  simulations  and  analytic  models  are  often  used  to  evaluate  test 
effectiveness.  The  Government  Program  Manager  must  evaluate  the  meaningfulness  and 
the  realism  of  the  "basis  for  testing." 
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Even  with  a  reasonably  large  sample  of  inserted  faults,  a  demonstration  can 
yield  only  limited  data  on  actual  test  effectiveness.  However,  a  demonstration  Is  also 
useful  in  validating  some  of  the  assumptions  and  models  that  were  used  during  the  earlier 
testability  analysis,  and  prediction  efforts  which  were  based  upon  a  much  larger  fault  set. 
If  certain  assumptions  or  models  are  invalidated  by  the  demonstration  or  T&E  activity, 
appropriate  portions  of  test  effectiveness  predictions  and  analyses  should  be  repeated 
and  new  predictions  should  be  made. 

The  Government  Program  Manager  must  also  evaluate  the  results  of  the  T&E 
activities.  The  degree  to  which  the  diagnostics  capability  achieved  the  specified 
requirements  must  be  determined.  Based  upon  this  degree  of  effectiveness,  it  may  be 
necessary  to  change  the  scope  or  basis  of  the  next  phase  of  testing. 

Diagnostics  DT&E  requirements  are  performed  in  accordance  with  the  System 

TEMP. 


The  major  approaches  of  DT&E  for  diagnostics  include  actions: 

0  To  proceed  in  phase  with  the  system  and  support  equipment  development, 
so  that  Built-In  Test  (BIT)  Is  tested  and  evaluated  concurrently  with  system 
performance;  BIT  and  System  Integration  Test  (SIT)  tested  and  evaluated 
concurrently  with  subsystem  integration  and  system  testing;  and,  system 
Integration  and  safety  testing  are  concurrent  with  diagnostic  testing  of  BIT 
and  SIT  features. 

o  To  implement  the  Diagnostics  Maturation  Program  so  that  deficiencies, 
ambiguities,  and  additional  failure  modes  identified  during  DT&E  are 
recorded  in  a  timely  manner  to  ensure  traceability  and  appropriate 
corrections  are  made  to  the  integrated  diagnostic  procedures. 

o  To  evaluate  embedded  diagnostic  design  as  a  separate  entity  in  order  to 
assure  that  it  has  been  incorporated  adequately  as  part  of  the  system 
design. 

o  To  evaluate  for  100%  diagnostic  capability  in  selected  critical  areas  of 
system  design  using  fault  insertion  techniques. 

o  To  analyze  the  system  design  hierarchy  of  test  tolerances  (e.g.,  between 
system  BIT  and  LRU  and  SRU-level  BIT)  in  order  to  minimize  false  alarms. 

o  To  complete  feasibility  DT&E  on  prototype  and  preproduction  units  in  order 
to  assess  technical  risks  and  develop  solutions  to  remedy  deficiencies. 
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During  FSD,  specific  diagnostic  capabiiity  segments  of  DT&E  efforts  include  the 
following  requirements: 

o  When  available,  ATE  shall  be  evaluated  for  initial  use  supporting  build  and 
check-out  of  systems.  Manual  procedures  and  associated  operational 
prototypes  shall  be  developed  for  support  of  test  activities. 

o  Engineering  evaluation  of  the  diagnostic  elements  capability  at  subsystem 
and  system  levels  shall  be  conducted  in  concert  with  system  integration 
testing  activities,  including  evaluation  tests  in  the  engineering  laboratory  and 
system  integration  test  facilities. 

o  Effective  development  of  a  diagnostic  capability  requires  that  testing  of 
diagnostic  capabilities  proceed  concurrently  with  prime  and  support 
equipment  development  in  an  orderly  and  planned  time-phased  manner. 
The  object  of  the  following  diagnostics  testing  approach  is  to  provide  a  viable 
Organizational-  and  Intermediate-level  diagnostic  capabiiity  for  use  in 
support  of  flight  and  operational  testing  activities  to  provide  for  early 
maturation  of  the  diagnostic  capability.  It  should  also  be  a  program  objective 
to  validate  the  diagnostic  capability,  as  well  as  initial  reliability  and 
maintainability  requirements  before  production. 

o  During  early  equipment  development  tests,  built-in  test  features  should  be 
tested  and  evaluated  concurrently  with  equipment  performance  testing.  BIT 
performance  is  just  as  essential  to  overall  weapon  system  performance  as 
the  usually  emphasized  aspects  of  equipment  performance.  Simulated 
equipment  failures  should  be  used  to  assist  in  BIT  testing  and  evaluation. 

o  As  equipment  progresses  to  subsystem  integration  and  performance  testing, 
BIT  and  System  Integrated  Test  (SIT)  features  should  be  concurrently 
tested,  evaluated,  and  corrected.  Simulated  equipment  failures  should  again 
be  used  for  BIT/SIT  testing  and  evaluation. 

0  System  integration  and  safe-for-flight  testing  of  equipment  should  include 
diagnostic  testing  of  BIT  and  SIT  features  to  assure  readiness  of  this 
capability  for  Flight  Test  Support.  Concurrently,  Organizational-level  support 
equipment  required  for  diagnostic  support  should  be  tested  to  enable  its  use 
in  the  test  program,  together  with  Preliminary  Maintenance  Manuals  for 
initial  Operational  Test  and  Evaluation.  Simulation  of  equipment  failures  to 
evaluate  diagnostic  capabilities  should  be  included  in  this  testing  effort. 

o  Qualification  testing  of  both  prime  and  support  equipment  shall  include 
validation  of  diagnostic  capability,  which  is  a  required  aspect  of  both 
equipment  and  system  performance.  Simulated  equipment  failures  should 
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I _ I 

be  included  in  the  diagnostic  validation  test  program.  Evaluation  of  BIT/SIT 
should  also  be  conducted  during  environmental  extreme  testing  of  the  prime 
equipment  and  support  equipment,  to  assure  its  proper  functioning 
throughout  the  required  equipment  performance  envelope. 
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CHECKLIST 


Does  the  contractor’s  Integrated  Test  Plan  provide 
adequate  detail  concerning  specific  T<ScE  procedures, 
data  bases,  models,  test  articles  and  scope  of  testing? 

E/  Have  critical  or  high  risk  items  related  to  diagnostic 
capability  been  identified  and  highlighted? 

Are  the  necessary  test  articles  available  to 
conduct  realistic,  timely  tests? 

^  Is  the  level  of  government  involvement  in  diagnostics 
DT<ScE  adequate  to  ensure  validity  of  tests  performed, 
results  documented,  data  accumulated? 

Ef  Is  there  a  direct  feedback  loop  in  the  engineering 

development  effort  to  deal  with  diagnostic  deficiencies? 

Has  every  opportunity  to  evaluate  diagnostics  during 
DT<ScE  activities  been  identified? 

^  Are  diagnostic  related  DT<ScE  activities  consistent  with 
the  Diagnostics  Maturation  Plan? 


OPERATIONAL  TEST  AND  EVALUATION 


REQUIREMENT  #6.3 


WEAPON 

SYSTEM 

ACQ.  PHASE 

OPER. 

REQMTS. 

CONCEPT 

EXPLOR. 

DEM/ 

VAL 

FSD 

PROD/ 

DEPL. 

WEAPON 

SYSTEM 

ACTIVITIES 

lOT&E  FOT&E 

DIAGNOSTIC 

ACTIVITIES 

A  A 

DIAGNOSTICS 

OT<ScE 

DIAGNOSTIC  ACTIVITY 


Evaluate  diagnostic  performance  characteristics  in  a  realistic  operational 
environment  during  Operational  Test  and  Evaluation  (OT&E)  activities  in  order  to 
determine  diagnostic  capabilities  achieved  and  to  identify  deficiencies  in  the  diagnostic 
capability.  Diagnostics  OT&E  should  focus  on  the  capability  achieved  by  the  integration  of 
the  various  planned  diagnostic  elements  into  a  comprehensive,  cohesive  diagnostics 
subsystem.  The  Government  Program  Manager  should  provide  assistance  to  the  OT&E 
organizations,  as  required. 

PROCEDURE 

Operational  Test  and  Evaluation  (OT&E)  is  the  field  test,  under  realistic 
conditions,  for  the  purpose  of  determining  the  effectiveness  and  suitability  of  the  system  or 
equipment  for  use  in  combat  by  typical  military  users;  and  the  evaluation  of  the  results  of 
such  tests. 

GUIDANCE 


Operational  Test  and  Evaluation  (OT&E)  activities  Include  Initial  OT&E  (lOT&E) 
and  Follow-on  OT&E  (FOT&E).  The  results  of  DT&E  activities  should  be  analyzed  by  the 
Government  Program  Manager  to  ensure  consistency  and  continuity  of  T&E  activities. 
Operational  Test  and  Evaluation  (OT&E)  must  bo  accomplished  prior  to  the  Milestone  III 
decision.  Diagnostics  OT&E  is  performed  to  provide  a  valid  estimate  of  the  operational 
effectiveness  and  suitability  of  the  system’s  integrated  diagnostics  design  and  procedures 
using  test  items  sufficiently  representative  of  the  expected  production  items. 


6-21 


OPERATIONAL  TEST  AND  EVALUATION 


REQUIREMENT  #6.3 


Major  approaches  to  diagnostics  OT&E  include: 
o  Testing  in  an  environment  as  operationally  realistic  as  possible 
o  OT&E  initiated  as  early  as  possible  during  the  FSD  Phase 
o  Testing  for  adherence  to  overall  OT&E  objectives,  with  respect  to  diagnostics 
o  Continued  coordination  with  the  Diagnostics  Maturation  Program 
o  Evaluation  for  1 00%  diagnostic  capability  in  selected  critical  areas 
o  Random  diagnostics  testing  in  noncritical  areas 

o  Further  analysis  of  test  tolerances  related  to  the  system  hierarchy  and 
embedded/extemal  diagnostic  procedures  in  order  to  minimize  false  alarms. 

Testing  (particularly  operational  tests)  and  data  collection  should  focus  on  the 
diagnostics  requirements.  Testing  and  data  collection  should  also  evaluate  the  specified 
parameters;  namely,  identification  of  critical  failures,  the  false  alarm  rate,  the  percentage  of 
faults  detected  and  isolated  automatically  or  manually,  associated  repair  times,  the 
unnecessary  removal  rate,  consistency  of  test  results,  and  the  adequacy  of  personnel 
skills  considering  all  maintenance  incidents. 

Use  of  the  diagnostic  capability  that  is  planned  for  field  maintenance  personnel 
should  be  required  whenever  there  is  a  need  for  system  maintenance.  This  applies  to 
maintenance  performed  by  either  the  contractor  or  the  user.  Thus  contractors  should  be 
required  to  use  the  diagnostic  capability  in  acceptance  and  qualification  tests  and  in  the 
manufacturing  and  quality  assurance  process  to  the  maximum  extent  possible.  In  addition 
to  contributing  to  the  maturation  of  the  diagnostic  capability,  it  is  anticipated  that  greater 
contractor  use  of  diagnostics  in  these  processes  could  result  in  production  cost  savings. 

The  diagnostic  capability  should  be  evaluated  with  respect  to  the  levels  planned 
and  set  forth  in  the  Diagnostics  Maturation  Plan.  Based  upon  the  difference  between 
planned  levels  of  capability  and  actual  levels  of  capability,  Diagnostics  T&E  and  corrective 
action  may  need  to  be  accelerated  or  adjusted.  (See  Requirement  #7.1  for  more 
Information.) 

During  OT&E,  system  performance,  operational  suitability  and  supportability 
factors  are  evaluated  in  an  operationally  realistic  environment.  There  are  two  types  of 
information  that  can  be  obtained  for  Diagnostics  T&E:  1)  faults  within  the  system  and  how 
those  faults  were  identified  (diagnosed);  and,  2)  faults/deficiencies  within  the  diagnostic 
capability.  For  the  latter,  this  includes  evaluation  of  each  element  which  contributes  to  the 
total  diagnostic  capability,  as  well  as  the  capability,  achieved  by  integration  of  the 
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diagnostics  eiements.  Focused,  detaiied  T&E  activities  discussed  in  Requirement  #  6.2 
should  be  continued.  The  former  type  of  data  can  be  obtained  as  a  result  of  Reliability 
Growth  Testing.  The  following  specific  information  should  be  evaluated  for  each  fault 
occurrence. 

1 .  How  did  the  failure  manifest  itself? 

2.  Was  the  manifestation  due  to  stressing  of  the  system  beyond  normal 
operational  limits? 

3.  If  a  BIT  alarm  occurred,  was  it  the  result  of  a  confirmed  failure? 

4.  What  techniques  were  used  to  isolate  the  fault? 

5.  How  long  did  fault  isolation  take  using  those  techniques? 

6.  Was  the  failure  mission  or  operation  critical? 

7.  Was  It  a  new  or  unplanned  failure  mode?  Was  BIT  supposed  to  detect  the 
♦allure?  Did  it? 

8.  Is  this  failure  mode  expected  to  be  encountered  in  the  operational  system? 

9.  Should  provisions  be  included  in  the  diagnostic  capability  to  deal  with  this 
failure  mode? 

10.  Will  this  involve  a  modification/addition  to  BIT?  ATE?  Manual  Test 
Equipment?  Maintenance  Procedures?  Skill  Levels?  Technical  Data? 
Maintenance  Aids? 

11.  Is  an  ECP  required? 

1 2.  Is  further  investigation  required? 

If  yes  -  What  plans  have  been  made? 

If  no  -  Why  not?  (brief  description) 

13.  Is  correction  of  the  diagnostic  deficiency  part  of  contractual  requirements? 
Tied  to  incentive  or  warranty  provisions? 
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CHECKLIST 


Are  diagnostics  OT<S£E  test  articles  sufficiently  repre¬ 
sentative  of  the  expected  production  items? 

O'  Is  the  diagnostics  OT<ScE  environment  as 
realistic  as  possible? 

of  Do  diagnostics  OT«S£E  plans  include  evaluation  for 
100%  diagnostic  capability  in  selected  critical  areas? 

O'  Do  OTScE  plans  Include  analysis  of  test 
tolerances  related  to  the  system  hierarchy 
and  off— line/on-line  diagnostics  procedures 
in  order  to  minimize  false  alarms? 

Is  diagnostics  evaluation  included  in  a  broad 
spectrum  of  OT&E  activities 
(e.g.,  Reliability  Growth  Testing)? 

Is  the  scope  of  diagnostics  0T<ScE  broad  enough 
to  realistically  do  a  preliminary  evaluation  of  the 
fielded  diagnostic  capability  provided  by  the 
combination  of  all  the  diagnostic  elements? 


MATURATION 


REQUIREMENT  §7 


MATURATION  OF  THE  DIAGNOSTIC  CAPABILITY 


OVERVIEW 


Historically,  often  a  weapon  system’s  diagnostic 
capability  does  not  meet  its  performance 
requirements  prior  to  deployment.  The  basic 
reason  for  this  is  that  all  faults  cannot  be  predicted 
and,  thus,  adjustment  of  the  diagnostic  capability 
is  required  during  the  first  few  years  after 
deployment.  Essentially,  this  requires  a  well- 
planned  maturation  period,  which  allows  for  the 
growth  of  the  diagnostic  capability.  Closely 
coupled  with  this  maturation  is  the  requirement  for 
collection  and  analysis  of  data  relating  to  the 
performance  of  this  diagnostic  capability,  both  in 
the  field  and  in  the  factory.  Care  must  be 
exercised  by  both  the  government  and  the 
contractor  to  assure  that  proper  and  detailed  data 
is  collected.  Early  planning  for  this  maturation 
period  is  a  must. 


TESTABILITY/ 

DIAGNOSTICS 


PROGRAMMATIC 


REQUIREMENTS 


DESIGN 


ASSESSMENT 


REVIEWS 


EVALUATION 


IMPORTANT  CONSIDERATIONS  TO  BE  ADDRESSED 


Reqmt. 

7.1  Require  that  a  detailed  Diagnostics  Maturation  Plan  be  prepared  early 
in  the  acquisition  process. 

7.2  Determine  the  diagnostic  data  collection  requirement  and  establish  a 
system  for  collection  and  analysis. 
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WEAPON 

SYSTEM 

ACQ.  PHASE 

OPER. 

REQMTS. 

CONCEPT 

EXPLOR. 

DEM/ 

VAL 

FSD 

PROD/ 

DEPL. 

WEAPON 

SYSTEM 

ACTIVITIES 

zx  zx 

SYSTEM  PDR 

SPEC. 

DIAGNOSTIC 

ACTIVITIES 

A  A 

PLAN  UPDATE 

DIAGNOSTIC  ACTIVITY 


Most  diagnostic  implementations,  no  matter  how  well  conceived,  require  a 
period  of  time  for  identification  of  problems  and  corrective  action  to  reach  specified 
performance  levels.  This  requirement  is  established  in  order  to  formalize  the  diagnostics 
maturation  and  to  allow  the  maturation  to  be  initiated  early  in  the  test  and  evaluation 
process.  This  requirement  is  initiated  early  so  that  early  identification,  tracking,  and 
correction  of  diagnostic  problems  is  achieved.  The  planning  for  this  activity  is  formalized 
by  the  development  of  a  Diagnostic  Maturation  Plan  or  other  appropriate  document. 

PROCEDURE 


Although  this  plan  is  prepared  primarily  by  the  prime  contractor,  it  is  essential 
that  the  Government  Program  Manager  closely  monitor  the  direction  and  the  results  of 
maturation  planning.  This  activity  results  in  a  strategy  for  diagnostics  maturation.  The 
Government  Program  Manager  must  ensure  that  this  strategy  is: 

1.  Comprehensive 

o  Across  all  diagnostic  elements 
o  Includes  the  integration  of  the  elements 

2.  Timely 

o  is  initiated  early  to  plan  for  the  required  resources  and  implement 
corrective  actions 

o  Maturation  is  completed  by  Milestone  IV,  per  DoD-lnstruction-5000.2 
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3.  Coordinated 

o  Includes  coordinated  activities  from  the  "ilities" 

0  Utilizes  standard  data  collection  systems 

4.  Cost  Effective 

o  Allows  data  collection  to  be  transferable  and  usable  by  government  (i.e., 
DT&E  and  production  test  data). 

GUIDANCE 


A  program  to  mature  the  diagnostic  capability  should  be  planned  for  the  early 
fielded  production  systems.  A  one-to-three-year  maturation  program  should  be  planned 
for  complex  weapon  systems  which  have  extensive  automatic  testing  capability.  For  major 
weapon  systems,  the  coordination  with  Milestone  IV,  Logistic  Readiness  and  Support 
Review  (DoD-lnstruction-5000.2)  is  essential.  This  program  should  include  provisions  for 
on-site  collection  of  diagnostic  performance  data,  with  engineering  follow-up  to  provide 
corrective  actions. 

The  plan  should  define  an  approach  and  methodology  to  assure  that  as 
development,  test  and  evaluation,  and  early  operational  use  of  the  system  progress, 
problems  presented  by  new  failure  modes,  test  voids,  ambiguities,  and  test  toierance 
difficulties  are  recognized  and  defined,  and  their  solutions  are  traceable  to  diagnostic 
software  and  manual  procedure  updates.  The  plan  should  recognize  that  such 
occurrences  are  expected  and  normal  and,  therefore,  should  concentrate  on  problem 
recognition,  definition,  and  correction,  with  appropriate  tracking  for  traceability. 

The  approach  and  methodology  defined  should  recognize  that  a  basic  element 
of  the  integrated  diagnostics  concept  is  identification  of  the  set  of  faults  which  are  known 
or  expected  to  occur.  The  methodology  should  provide  for  definition  of  this  set,  initially 
through  Failure  Modes  and  Effects  Analysis,  Testability  Analysis,  and  other  tools  and 
experience.  Provision  for  growth  of  this  set,  as  new  failure  modes  are  encountered  during 
testing  and  deployment,  should  be  incorporated  in  the  plan,  together  with  explicit  criteria  to 
be  used  in  deciding  whether  or  not  a  newly  encountered  fault  should  be  added  to  the  set 
of  faults  for  which  explicit  diagnostic  procedures  (as  opposed  to  more  general  procedures) 
are  provided  for  detection  and  isolation  of  the  fault.  The  life  cycle  cost  effectiveness  of 
adding  explicit  diagnostic  procedures  for  the  newly  encountered  fault  should  be  one  factor 
considered  in  the  decision. 

The  plan  should  provide  for  an  orderly  development  and  maturation  process  for 
diagnostic  software  and  manual  procedures  throughout  the  development,  test  and 
evaluation,  and  early  operational  use  time  periods  of  the  weapon  system  and  its 
subsystems.  Methodology  to  assure  timely  and  continuing  technical  support  for  this 
maturation  process  by  both  contractor  and  government  activities,  with  a  minimum  of 
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administrative  delays,  should  be  a  feature  of  the  plan.  Orderly  transition  of  technical 
responsibilities  from  the  contractor  to  the  government  should  also  be  addressed. 

The  plan  should  present  milestones  to  be  met.  This  will  assure  that  the  final 
system  achieves  the  required  degree  of  diagnostic  capabiiity.  The  plan  should  show  the 
time  phasing  of  each  task  and  its  interrelationship  with  other  tasks.  It  should  identify 
required  data  review,  verification,  and  utilization  to  accomplish  the  required  tasks  and  to 
report  progress,  problems,  and  tradeoffs.  The  plan  should  assure  the  proper 
implementation  of  diagnostic  design  features  by  designers  and  subcontractors. 

This  plan  will  enable  the  procuring  activity  to  monitor  and  evaluate  the 
contractor’s  progress  toward  achieving  the  required  diagnostic  capability  through  the 
system  design,  development,  test,  and  evaluation  process.  The  government  may 
establish  contractual  incentives  for  appropriate  milestones  throughout  the  diagnostic 
development,  test,  and  evaluation  process.  Milestones  selected  should  include 
completion  of  design  for  testability  assessment  and  other  diagnostic  system  design 
assessments;  compietion  of  diagnostic  test  element  and  diagnostic  system  evaluations,  in 
concert  with  equipment  design  evaluation  testing  at  the  LRU/subsystem  level;  and 
diagnostic  system  testing,  in  concert  with  systems  integration  test  facilities  and  during  the 
operational  test  program.  The  plan  should  also  provide  for  government  evaluation  and 
final  acceptance  of  the  automatic  test  programs  and  manual  troubleshooting  procedures  in 
maintenance  technical  publications. 

During  the  Dem/Vai  Phase,  maturation  planning  is  centered  on  preliminary 
planning  for  data  collection,  analysis  and  coordination  with  similar  requirements  for 
reliability,  maintainability,  logistics,  data  collection,  analysis  systems,  etc.  Specifically,  this 
planning  should  identify  potential  data  sources,  such  as: 

o  Laboratory  testing 
o  Developmental  testing 
o  Operational  test  and  evaluation 
o  Acceptance  testing 
o  Preproduction  testing 
o  Production  testing 
o  Operation. 

The  requirement  for  diagnostic  data  collection  should  be  coordinated  with  similar 
requirements,  such  as: 

MIL-STD-785 

Task  104  -  Failure  Reporting  Analysis  and  Corrective  Action  (FRACAS) 

Task  105  -  Failure  Review  Board 

Task  301  -  Environmental  Stress  Screening 

Task  302  -  Reliability  Development/Growth  Test  (RDGT) 
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MIL-STD-470 

Task  104  -  Data  Collection,  Analysis,  and  Corrective  Action 

MIL-STD-471 

Maintainability  Verification/Demonstration/Evaluation 
MIL-STD- 1388-1 

Task  501  -  Supportability  Test,  Evaluation,  and  Verification 

MIL-STD-781 

Reliability  Testing  For  Engineering  Development, Qualification  and 
Productfon 

MIL-STD-2155 

Failure  Reporting,  Analysis,  and  Corrective  Action  System 
MIL-STD-2165 

Task  103  -  Testability  Data  Collection  and  Analysis  Planning 

No  standard  format  for  the  Diagnostics  Maturation  Plan  exists.  The  plan  may  be 
incorporated  in  another  plan,  such  as  the  SEMP.  The  guidance  provided  above  should  be 
completed  with  the  data  collection  and  feedback  system  (Requirement  #7.2)  established  in 
the  preparation  of  this  plan. 
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MATURATION  PLANNING 


REQUIREMENT  #7.1 


CHECKLIST 

□r  Does  the  Diagnostic  Maturation  Plan  include  a  strategy 
for  the  collection  of  diagnostic  performance  data 
through  DT<ScE,  OT<ScE,  Production,  Initial  Operational 
Use,  and  Deployment? 

EZ'  Are  the  data  sets  collected  throughout  the  above 
periods  realistic,  comprehensive,  and  timely?  Is  there 
a  logical  flow  through  the  data  sets  to  be  collected  so 
that,  as  maturation  proceeds,  data  from  one  period  can 
be  logically  used  in  the  next? 

Does  the  plan  include  provisions  for  all  diagnostic 

elements  —  embedded  and  external  — 

as  well  as  the  integration  of  the  diagnostic  elements? 

Is  the  integration  of  the  diagnostic  elements  planned 
for  early  enough  to  allow  evaluation  and  cost-effective 
corrective  action  (e.g.,  prior  to  production  go-ahead)? 

O'  Do  the  data  sets  to  be  collected  allow  for 
government  capture  and  use  of  the  data? 

EZf  Are  standard  government  data  collection  systems 
utilized? 

^  Is  there  an  appropriate  degree  of  coordination  with 
similar  requirements  (e.g.,  if  Reliability  Growth 
Testing  is  a  program  requirement,  is  data  from  that 
testing  planned  to  be  coordinated  with  Diagnostics 
Maturation)? 

of  Has  the  prime  contractor  included  provisions  and  plans 
for  dealing  with  subcontractors  and  vendors  in  the 
Diagnostic  Maturation  Plan? 

Are  configuration  control  requirements  included  in  the 
maturation  planning  for  the  prime  system,  as  well  as 
for  the  diagnostic  elements? 


MATURATION  PU^NNING 


REQUIREMENT  #7.1 


CHECKLIST  (cont.) 

Does  Maturation  Planning  include  provisions  for  both: 

1 .  Adequacy  of  the  diagnostic  elements,  with  respect 
to  the  specified  allocated  capability,  and 

2.  Unplanned  failure  modes,  which  may  arise  throughout 
OT<ScE,  DT<ScE,  Production  Test,  and  Field  Use  Test. 

O'  Is  a  structured,  closed— loop,  analytic  process  planned 
for  the  resolution  of  any/all  deficiencies? 

of  Is  it  clearly  laid  out  who  is  responsible  for  what 
throughout  the  various  periods  of  the  maturation 
process  (i.e.,  who  (government  or  contractor)  is 
responsible  for:  (1)  data  collection;  (2)  analysis; 
and  (3)  corrective  action  through  DT&:E,  OT<ScE,  Produc¬ 
tion  Test  and  Field  Use  Test). 


OTiScE,  Produc- 


DATA  COLLECTION  AND  FEEDBACK 


REQUIREMENT  #7.2 


WEAPON 

SYSTEM 

ACQ.  PHASE 

CONCEPT 

EXPLOR. 

DEM/ 

VAL 

FSD 

PROD/ 

DEPL. 

WEAPON 

SYSTEM 

ACTIVITY 

ZS  ^  A 

SYSTEM  PRODUCT  ECP 

FABRICATION  BASELINE 

DIAGNOSTIC 

ACTIVITIES 

A  A  A  A  A 

DT&E  lOT&E  roi&E  DATA  COR- 

COL-  RECnVE 

LECTION  ACTION 

DIAGNOSTIC  ACTIVITY 


Data  relating  to  the  performance  and  effectiveness  of  the  diagnostic  capability 
must  be  collected  during  development,  production,  and  operation.  This  data  is  used  as 
the  basis  for  the  evaluation  of  diagnostics  and  for  the  correction  of  deficiencies. 

The  key  thrust  of  this  activity  is  definition  of  appropriate  data  to  be  collected, 
maximum  use  of  data  collected,  coordination  of  data  collection  systems,  and  a  structured 
approach  to  corrective  action. 

PROCEDURE 


This  activity  is  described  in  the  Diagnostic  Maturation  Plan.  The  Government 
Program  Manager  must  closely  monitor  and  evaluate  the  prime  contractor’s  planning  and 
implementation  of  diagnostic  data  collection  arid  feedback  requirements.  Issues  which 
must  be  monitored  include:  timing,  data  sets,  coordination  with  similar  requirements,  and 
a  structured,  closed-loop  approach  to  data  analysis  and  corrective  action. 

The  earlier  diagnostic  performance  deficiencies  are  identified,  the  sooner  a 
more  cost-effective  solution  can  be  implemented.  Therefore,  diagnostic  data  collection 
and  feedback  is  initiated  early  in  the  test  and  evaluation  process,  continues  through 
production  test,  and  extends  into  the  operational  environment.  Throughout  these  phases, 
different  types  of  data  are  collected,  different  data  collection  procedures  and 
methodologies  are  used,  and  different  types  of  analysis  techniques  are  conducted. 
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GUIDANCE 


There  are  no  standard  methods  for  data  collection  and  analysis.  As  indicated 
under  Requirement  7.1,  Maturation  Planning,  the  collection  of  this  type  of  data  is 
controlled  by  a  number  of  military  standards.  The  requirements  for  the  standards  which 
deal  with  logistics,  reliability,  maintainability,  testability,  human  engineering,  and  safety 
overlap  one  another  (many  times  data  required  by  one  may,  indeed,  be  required  by  the 
other(s)).  Thus  close  coordination  among  these  various  data  requirements  is  needed.  A 
single  data  base  is  desirable. 

In  addition,  these  data  systems  are  required  during  system  development,  as  well 
as  after  the  system  is  deployed.  There  must  be  compatibility  between  the  contractor's 
data  system  and  the  follow-on  government  data  system,  so  that  traceability  exists  from 
cradle-to-grave. 

The  data  collection  procedures  closely  follow  the  test  and  evaluation  functions. 
As  explained  in  DoD  Directive  5000.3,  Test  and  Evaluation,  the  time  periods  and 
sequences  for  Development  Test  and  Evaluation  and  Operational  Test  and  Evaluation 
vary  from  program-to-program.  They  can  overlap  and  even  be  done  as  a  combined  test 
and  evaluation.  Thus  there  are  no  standard  guidelines  that  specify  the  exact  points  in  the 
weapon  system  acquisition  phase  where  data  is  to  be  collected.  The  system  must  be 
flexible  to  incorporate  data  as  data  is  generated. 

The  Government  Program  Manager  should  ensure  that  the  proper  data  is 
collected  and  that  corrective  actions  are  pursued.  Care  must  be  taken  to  collect  only  that 
data  required  to  assure  that  the  diagnostic  capabilities  are  performing  as  required. 
Automated  data  collection  systems  can  be  employed.  Usually  these  are  more  effective,  as 
they  are  less  dependent  on  human  motivation  to  supply  the  required  information. 

Corrective  analysis  and  actions  should  be  in  a  closed-loop  system,  so  that  each 
deficiency  identified  remains  an  open  item  until  it  is  formally  documented  as  being 
corrected. 


The  data  collection  and  feedback  system  should  be  designed  so  that  specific 
information  is  collected  on  the  performance  of  the  entire  diagnostic  capability,  as  well  as 
for  each  of  the  diagnostic  elements  that  make  up  the  diagnostic  capability.  The 
information  must  be  collected  in  quantitative  form,  if  possible,  and  related  to  System 
Specification  requirements.  Thus  the  following  guidelines  on  the  type  of  data  to  be 
collected  need  to  be  tailored  so  that  the  information  can  be  related  to  System  Specification 
requirements  and  so  that  it  is  clearly  apparent  who  is  to  supply  the  information  and  when 
this  information  is  to  be  supplied.  Examples  of  the  type  of  data  to  be  collected  follow. 
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DATA  COLLECTION  AND  FEEDBACK  REQUIREMENT  #7.2 

Diagnostic  Data  Feedback 

o  Effectiveness  and  efficiency  of  each  diagnostic  element 

o  Effectiveness  and  efficiency  of  the  diagnostic  elements  as  an  integrated 
system 

o  Operational/support  impact  of  the  diagnostic  deficiencies 
o  Corrective  actlon(s)  which  should  be  taken  or  have  been  taken. 

BiT  Effectiveness 

o  Fault  isolation  time. 

Tracking  of  Faise  Aiarms 
o  Type  of  alarm 

0  Frequency  of  alarm  occurrence 
o  Cause  (if  known) 

o  Potential  consequences  of  ignoring  the  alarm  (crew  safety,  mission 
reliability) 

o  Operational  costs  of  responding  to  false  alarms  (aborted  missions,  degraded 
mode  operation,  system  down  time) 

o  Support  costs  associated  with  the  false  alarm 

o  Operational  environment  when  alarm  occurred. 

ATE  Effectiveness  Feedback 

o  Workarounds  required  to  overcome  mechanical  or  electrical  deficiencies  in 
the  UUT/ATE  interface 

o  Consistency  of  ATE  FD  results  with  initial  BIT  indications 
o  Repeatability  of  ATE  test  results 
o  Ambiguity  size 
o  Fault  isolatiori  time. 
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Integration  of  Diagnostic  Elements 


REQUIREMENT  #7.2 


o  Consistency  of  diagnostic  resources  with  the  training/skill  levels  of  assigned 
personnel 

o  Effect  of  false  alarms  and  unnecessary  removals  on  operational  availability 
and  maintenance  workload 

o  Shop  throughput 

o  Adequacy  of  technical  information 

o  Logistic  delay  time 

o  BIT  reliability 

o  ATE  reliability. 


Diagnostic  data  collection  and  diagnostic  capability  performance  assessment 
may  lead  to  the  requirement  for  corrective  action.  Corrective  action  may  Involve  redesign 
of  prime  equipment,  test  equipment,  interface  devices,  maintenance  documentation,  built- 
in  test  circuits,  diagnostic  software,  and  ATE  test  programs.  All  changes  must  be  made 
under  strict  configuration  control. 
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REQUIREMENT  #7.2 


CHECKLIST 

Has  the  data  collection  system  been  designed  to  collect 
only  the  required  data?  In  quantitative  terms? 

Are  the  government,  contractor,  and  subcontractor  data 
collection  systems  compatible? 

O'  Is  there  direct  communication  back  and  forth  between 
the  person  who  reports  a  problem  and  the  person  who  is 
responsible  for  correcting  the  problem? 

O'  Will  all  known  failures  be  reported? 

O'  Will  all  failures  be  analyzed  to  sufficient  depth  to  iden¬ 
tify  failure  causes  and  necessary  corrective  actions? 

O'  Will  all  failure  analysis  reports  be  closed  out  within 

30  days  of  failure  occurrence  or  rationale  provided  for 
any  extensions? 

Ei  Will  corporate  management  be  automatically  alerted  to 
failures  exceeding  close-out  criteria? 
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LESSONS  LEARNED 
DESIGNING  A  NEW  SYSTEM 


I.  INTRODUCTION 

A  young  Air  Force  technician  assigned  to  maintain  one  of  today’s  sophisticated 
weapon  systems  is  on  his  way  to  another  day  of  work.  As  he  traveis  his  prescribed  route, 
he  refiects  on  the  intricacies  of  the  equipment  with  which  he  works.  He  sighs  in 
amazement  regarding  how  easiiy  and  accurately  he  knows  what  to  replace  when  things  go 
wrong. 


Shortly  after  arriving  at  his  duty  station,  he  enters  his  code  at  a  computer 
terminal  and  is  provided  with  a  work  order  for  his  first  task  of  the  day.  The  work  order 
concerns  a  malfunction  which  was  detected  during  a  flight  oompleted  just  one-half  hour 
earlier.  A  quick  glance  at  the  work  order  reveals  which  system  failed,  what  time  it 
occurred,  and  the  Line  Replaceable  Unit  (LRU)  which  is  to  be  replaced  to  correct  the 
problem.  After  a  quick  trip  to  a  supply  point  for  a  serviceable  LRU,  with  tool  box  and 
checklist  in  hand,  he  departs  for  the  flight  line.  The  defective  LRU  is  changed  within 
minutes  after  his  arrival.  A  quick  operational  check,  using  the  checklist  and  on-board  test 
system,  confirms  that  no  other  failures  have  occurred,  and  the  system  is  declared 
operational. 

Back  at  the  Intermediate  shop,  the  flight  line  portion  of  the  work  order  is  closed. 
This  is  quickly  done,  with  a  minimal  amount  of  information  input  at  the  computer  terminal 
regarding  the  work  accomplished.  The  defective  LRU  is  placed  on  its  corresponding 
Automatic  Test  Equipment  (ATE).  Keys  within  the  LRU  provide  identification  information 
to  the  computer  contained  within  the  ATE.  Failure  conditions  and  symptoms  recorded  on- 
aircraft  at  the  time  of  the  failure  are  also  transferred  to  the  ATE  computer  via  the  computer 
network.  The  ATE  rapidly  goes  through  a  set  of  tests  specifically  tailored  to  the  reported 
failure  conditions  and  the  failed  single  component  is  identified. 

The  failed  component  is  replaced  with  a  new  component  obtained  from  the 
bench  stock  located  within  the  shop.  After  being  checked  again  with  the  ATE  to  verify 
serviceability,  the  LRU  is  given  a  quality  control  inspection  and  returned  to  the  supply 
point,  where  it  once  again  becomes  a  serviceable  asset. 

The  above  scenario  (or  parts  thereof)  has  been  a  goal  of  military  services  for 
many  years.  Great  strides  have  been  made  in  diagnostics  toward  its  achievement,  yet 
even  total  success  in  limited  areas  does  not  lie  immediately  at  hand. 
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The  reasons  that  success  is  fleeting  are  many.  They  include  budget 
constraints,  a  relative  lack  of  importance,  political  considerations,  time,  and  the  complexity 
of  the  task--just  to  mention  a  few.  This  appendix  presents  a  few  glimpses  of  diagnostic 
activity  on  recent  programs,  results  obtained,  and  lessons  learned. 

The  Information  presented  is  a  composite  of  program  experiences  derived  from 
the  B-1B  and  F-1 1 1  aircraft,  as  well  as  the  Minuteman,  Peacekeeper,  and  Small  ICBM 
Strategic  Missiles. 

A.  Government  Program  Manager  Concerns 

Experience  has  shown  that  Government  Program  Manager  difficulties  in 
obtaining  good  diagnostics  lie  in  two  main  areas.  The  first  is  succeeding  within  man- 
derived  constraints,  which  limit  the  emphasis,  the  amount  of  resources  which  can  be 
applied,  and  program  efficiency  related  to  diagnostics.  The  second  major  area  of  concern 
is  the  technical  difficulties  involved.  Specifically  for  the  Government  Program  Manager, 
this  relates  to  the  identification  of  real  and  cost-effective  requirements  and  assuring  their 
achievement  in  the  hardware/support  system  design. 

Both  of  the  above  areas  are  demonstrated  in  the  following  pages  of  this 
document. 

B.  Time  Frame 

This  appendix  presents  lessons  learned  information  from  all  phases  of  an 
equipment’s  life  cycle.  In  consonance  with  the  scope  of  the  Program  Managers  Guide,  to 
which  it  is  attached.  As  such,  it  describes  activities  which  may  have  taken  place  over  the 
last  20  years,  or  more,  even  though  we  are  mainly  addressing  recently  deployed  weapon 
systems. 

C.  Constraints,  I.  e.,  Time,  Money,  and  Political 

The  lessons  learned  described  in  this  document,  in  almost  every  instance,  have 
something  to  do  with  trying  to  succeed  within  constraints  imposed  on  the  program  due  to 
limited  resources  and  other  political  considerations.  The  B-1  Aircraft  Program  is  an 
excellent  example  of  the  usual  limited  resource  problems,  compounded  by  political 
decisions  not  In  the  best  interest  of  the  program.  The  original  B-1  A  development  effort 
was  accomplished  from  1971  through  1977,  at  which  time  the  program  was  canceled. 
Four-and-one-half  years  later,  in  March  1982,  the  program  was  reinstated.  In  order  to 
make  up  for  lost  time,  a  concurrent  Full-Scale  Development  and  Production  Program  for 
the  B-1  B  was  contracted.  The  initial  expectations  were  that  the  earlier  B-1  A  effort  would 
allow  for  this  concurrence.  The  reality  was  that  there  were  hardware  changes  and  other 
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development  efforts  required  which,  if  time  had  permitted  prior  to  production,  wouid  have 
resulted  in  fewer  production  problems  and  better  diagnostics  at  first  deployment. 

II.  DEVELOPING/ESTABLISHING  REQUIREMENTS 

What  is  specified  in  the  procurement  specification  and  the  contractual  Statement 
of  Work  is  what  the  government  expects  to  receive.  In  the  area  of  diagnostics,  the 
government  experience  on  past  programs  has  not  been  the  best.  Systems  have  been 
introduced  into  the  inventory  with  diagnostics  that  have  proven  to  be  incomplete,  unable  to 
test  to  the  desired  level,  or  simply  do  not  operate  as  advertised.  The  basic  foundation 
upon  which  ail  other  successes  or  failures  are  directly  dependent  is  the  clear 
understanding  of  the  actual  requirements. 

A.  Structuring  Need  Statements  and  Work  Statements 

All  of  the  programs  surveyed  in  the  preparation  of  these  documents  seem  to 
have  an  item  in  common  dealing  with  their  diagnostic  requirements.  That  commonality 
factor  is  that  the  quantitative  diagnostic  requirements  imposed  are  derived  without  a  great 
deal  of  thought  and  analysis.  Typically,  diagnostic  requirements  are  more  what  has  been 
judged  by  someone  to  be  realistic  values,  rather  than  a  product  of  the  various  studies 
performed  to  determine  these  requirements. 

DoD-lnstruction-5000.2  and  other  related  documents  describe  a  structured 
acquisition  process  beginning  with,  among  other  things,  the  development  of  a  Mission 
Area  Analysis  and  a  Mission  Need  Statement,  included  in  the  Mission  Need  Statement  is 
a  discussion  of  the  Mission  and  Threat,  Alternative  Concepts,  and  Technology 
involvement.  Subsequently,  during  the  Concept  Exploration  Phase,  studies  are  conducted 
to  develop  a  System  Concept  Paper  which  more  thoroughly  defines  possible  alternatives, 
and  a  selected  concept.  Many  items  are  taken  under  consideration  during  this  time  frame 
including  readiness,  maintainability,  manpower  and  training. 

It  is  this  process  which  generally  develops  the  procurement  specifications.  It  is 
this  process  which  should  develop  the  diagnostic  specifications.  The  initial  document,  the 
Mission  Need  Statement,  is  the  basic  source  on  which  these  specifications  are  developed 
and  is  most  important  to  the  process. 

B.  Developing  Cost-Effective  Requirements 

Specifications  both  greater  than  and  less  than  those  actually  required  to  meet 
the  end  mission  generally  cause  unnecessary  costs.  Excessive  specifications  increase 
costs  in  every  acquisition  phase,  i.  e.,  development  procurement,  and  support. 
Specification  values  less  than  required,  cause  excessive  costs  .mainly  in  the  operational 
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phase.  These  come  from  developing  workarounds,  not  meeting  mission  needs,  and 
driving  more  assets  into  the  repair  process. 

MIL-STD-1 388-1  (Logistic  Support  Analysis)  describes  specific  tasks  to  be 
performed  to  develop  those  considerations.  These  tasks  include  a  comparative  analysis, 
consideration  of  standardization,  investigating  technological  opportunities,  and  trade-off 
analysis.  This  process  is  geared  to  the  development  of  specifications. 

This  standard,  written  in  1 983,  was  not  available  to  be  imposed  on  any  of  the 
programs  covered  by  this  appendix.  Judgmentally,  however,  from  the  experiences  on  the 
covered  programs,  it  is  a  consensus  that  the  MIL-STD-1 388-1 A  approach  should  be 
conscientiously  imposed  to  develop  good,  well-founded  diagnostic  requirements  and 
program  plans. 

C.  Adequate  Specification  Definitions 

Determining  the  proper  diagnostic  specifications  necessary  to  meet  the  mission 
need  is  one  thing.  Describing  them  in  such  a  way  that  they  will  be  interpreted  properly  is 
another. 


The  following  Is  one  of  the  diagnostic  requirements  imposed  on  the  B-1B.  The 
Central  Integrated  Test  System  (CITS)  shall  provide  an  assurance  of  95%  to  the  aircrew 
that  the  system  performance  is  operationally  acceptable  or  that  the  indicated  failure  is 
valid  during  in-flight  performance  and  ground  readiness  tests.  The  CITS  shall  provide  fault 
isolation  to  an  LRU,  with  a  certainty  of  at  least  75%  in  the  ground  fault  isolation  mode." 

Another  requirement  stated  that  "false  alarms  could  not  exceed  2%."  Both 
seemingly  good  requirements,  but  two  problems  ensued  in  their  accomplishment.  First 
and  foremost  was  the  problem  in  the  definition  of  the  percent  base.  Percentages  are  often 
used  in  defining  requirements.  But  when  so  used,  it  must  be  stated  as  a  percent  of  what. 
False  alarms,  as  a  percent  of  the  possible  alarms,  give  one  result.  False  alarms,  as  a 
percent  of  the  number  of  total  alarms  indicated,  give  another.  When  written,  one  must 
assume  that  achievement  based  on  the  definition  of  the  writer  would  meet  mission  needs. 
In  reality,  when  achieved  based  on  a  legal  and  implied  definition,  the  results  were  far  from 
those  required  by  the  operational  command. 

A  second  problem,  but  in  this  case  a  lesser  problem,  was  a  conflict  between  the 
requirements.  The  first  requirement  above  allows  a  5%  false  alarm  rate  (100  minus  the 
95%  accuracy).  The  second  allows  only  2%.  Specification  ambiguity  leads  to 
interpretation  which  will  not  necessarily  end  with  the  desired  result. 
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III.  STRUCTURING  DESIGN  CONCEPTS/CONSTRAINTS 

A.  Controlling  vs.  Constraining  the  Contractor 

Today’s  trend  in  specification  and  contractor  direction  is  to  provide  the 
contractor  with  the  maximum  leeway  in  meeting  specified  requirements.  The  objective  is 
to  ailow  the  contractor  to  define  aiternatives,  select  from  the  alternatives  those  which  he 
can  best  implement,  and  provide  a  product  which  meets  all  of  the  requirements. 

Existing  systems  covered  by  this  document  were  all  developed  under  a  more 
structured  specification  approach.  The  previous  school  of  thought  was,  generally 
speaking,  that  the  more  things  which  can  be  controlled  by  the  specifications,  the  more 
chance  the  end  product  will  be  produced  as  desired.  Experience  with  that  approach  has 
led  to  the  more  open  trend.  This  is  because  the  tighter  approach  did  not  allow  the 
contractor  to  make  maximum  use  of  his  many  possible  alternatives. 

Good,  easy,  and  frequent  communications  between  the  customer  and 
contractor  diagnostics  personnel  is  a  must.  A  specific  item,  noted  almost  unanimously 
with  B-1B  personnel,  is  the  Importance  of  the  function  provided  on  that  program  by 
devoted  diagnostic  managers  in  both  the  Air  Force  and  customer  organizations.  The 
shortfall,  however,  was  that  these  managers  did  change  from  time-to-time.  On  long 
programs  this  is  to  be  expected.  The  lesson  to  Government  Program  Managers  is  to 
provide  for  these  changes.  Close  coordination  with  potential  backups  and  well- 
documented  decisions  will  minimize  upsets  due  to  personnel  changes. 

Another  important  point  noted  was  that  these  diagnostic  managers  need  some 
very  special  skills  besides  those  of  a  skillful  manager.  Each  should  be  an  experienced 
engineer,  with  a  thorough  understanding  of  the  hardware  involved,  diagnostic  methods, 
and  support  concepts. 

B.  Establishing  and  Designing  to  the  Maintenance  Concept 

The  logistic  support  analysis  tasks  of  MIL-STD- 1388-1  which  are  concerned  with 
the  development  of  maintenance  concepts  and  constraints  are  very  important  for  the 
diagnostics  community.  The  Government  Program  Manager  should  ensure  that  these 
tasks  are  completed,  as  appropriate,  and  that  the  resulting  maintenance  concept  is  well 
understood  by  all.  The  MIL-STD-1 388-1  tasks  are  structured  to  ensure  consideration  of 
existing  resources,  compatibility  with  deployment  and  operational  requirements,  and  the 
use  of  trade  studies. 

The  tie  between  the  diagnostic  method  and  the  maintenance  concept  is 
bidirectional.  They  need  to  be  established  in  unison.  The  maintenance  concept  is 
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developed  based  on  expected  diagnostic  capabilities.  The  diagnostic  design  ultimately 
forces  the  real  maintenance  concept. 

Established  Air  Force  maintenance  policies  generally  utilize  system  operation  as 
the  final  determinant  of  the  need  for  maintenance,  if  the  system  is  functioning  within 
tolerance,  don’t  fix  it.  A  unique  situation  has  developed  on  the  B-1B.  Due  to 
redundancies  designed  in  the  systems,  overall  operation  appears  to  be  normal,  while 
some  specific  parts  thereof  are  not  functioning  as  they  should.  The  diagnostics  says 
these  parts  should  be  replaced.  System  operation  appears  to  indicate  everything  is  OK. 
To  date,  partially  due  to  the  lack  of  confidence  in  the  aircraft  diagnostics  and  partially  due 
to  established  habits,  often  these  type  malfunctions  are  not  being  repaired.  Generally  the 
diagnostics  is  believed  to  be  faulty  and  no  maintenance  action  is  taken  until  something 
else  happens  to  make  the  system  inoperative. 

This  experience  shows  that  changing  existing  practices  is  slow.  If  it  is  confused 
with  the  lack  of  confidence  in  the  diagnostics,  the  change  is  even  more  difficult. 

The  logistics  manager,  ATE  manager,  and  automatic  test  equipment  designer 
are  all  vital  elements  in  determining  what  off-equipment  testing  is  required.  Once  the 
option  for  automated  testing  is  confirmed,  the  ATE  designers  must  convince  the  Unit 
Under  Test  (UUT)  designer  to  incorporate  "design  to"  criteria  for  maintainability,  reliability, 
and  testability.  Care  must  be  taken  to  define  the  need  for  ATE,  how  the  ATE  is  to  be 
used,  how  the  UUT  will  be  designed  for  built-in  test,  and  interfacing  abilities.  ATE 
effectiveness  Is  directly  and  immediately  dependent  on  this  codevelopment  with  the  UUT. 

IV.  Meaningful  Prediction  and  Assessment  Methodology 

In-process  assessment  of  diagnostics  achievements  has,  in  the  past,  been  less 
than  adequate.  In  fact,  one  of  the  most  definitive  and  often  repeated  B-1 B  lessons  is  the 
need  for  an  operational  period  to  mature  the  diagnostic  design.  That  lesson  is  described 
in  paragraph  VII  below.  Prediction  and  assessment  techniques  have,  in  the  past,  failed  to 
provide  ■sufficient  information  to  uncover  all  of  the  inadequacies  and  shortcomings. 
Significant  emphasis  is  currently  being  placed  on  testability  analysis,  reliability,  and 
maintainability  assessment  tools  under  the  umbrella  of  Computer-Aided  Acquisition  and 
Logistics  Support  (CALS).  With  that  emphasis,  one  should  expect  great  improvements  in 
assessment  techniques.  The  point  for  the  Government  Program  Manager  on  this  subject 
is  to  ensure  that  predictions  and  assessments  performed  are  sufficient  to  demonstrate  that 
diagnostic  requirements  are  being  fulfilled. 

A.  Methodology 

The  CALS  initiative  would  include  diagnostics  design  as  an  integral  part  of  the 
CAE/CAD  design.  The  concept  is  that  rules  and  techniques  would  be  established  in  the 
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computer  workstation.  As  a  specific  item  is  designed,  it  is  constantly  checked  for  test 
access,  built-in  test  capability,  and  other  established  rules. 

This  concept  works  fine  for  evaluating  the  diagnostic  characteristics  of  a  single 
electronic  assembly.  Evaluation  of  a  weapon  system’s  central  test  system  is  another 
question.  For  the  B-1B,  a  complete  integration  lab  was  developed  to  test  the  diagnostics 
software  in  a  functional  environment.  That  process  was  useful,  but  still  under  the  best  of 
lab  conditions  some  things  cannot  be  developed  to  the  optimum  level.  An  excellent 
example  is  the  philosophy  for  checking  the  thrust  of  a  jet  engine.  Simulated  lab  conditions 
equate  more  to  an  aircraft  being  on  the  ground  where  thrust  is  compared  to  a  reference 
schedule  of  gross  thrust  versus  turbine  blade  temperature  at  two  discrete  operating  points. 
These  two  operating  points  are  the  intermediate  and  maximum  power  setting.  To  develop 
an  in-flight  thrust  check,  a  reference  has  to  be  calculated  to  monitor  performance  across 
the  entire  power  range.  This  reference  is  obtained  by  comparing  the  engines  in 
synchronization  to  one  another  in  flight.  This  reference  requirement,  plus  many 
preconditions  necessary  for  calculating  or  examining  thrust,  dictates  actual  flight  testing  for 
development  of  a  valid  check. 

B.  Review  and  Feedback  Structure 

Time  and  management  emphasis  are  both  needed  to  assure  that  the  design 
benefits  from  the  assessment  process.  Logically,  one  does  not  need  a  whole  lot  of 
experience  to  understand  this.  However,  it  was  proved  once  again  on  the  B-1B  aircraft. 
Concurrent  Full-Scale  Development  and  Production  meant  that  funding  for  studies  and 
analysis  occurred  so  late  that  the  results  could  not  be  implemented. 

Management  direction  is  also  needed  or  results  will  go  unheeded.  If  the 
decision  is  between  getting  an  aircraft  into  the  air  on  schedule  or  improving  the  diagnostic 
capability,  what  will  the  Program  Manager's  decision  be? 

V.  DESIGN  REVIEWS 

Formal  Design  Reviews  provide  the  opportunity  for  the  customer  to  accept  what 
the  contractor  is  offering  or  insist  on  improvements.  If  the  contractor  can  demonstrate  that 
he  is  meeting  the  specifications,  the  customer  can  ask  no  more.  At  this  point,  it  is  the  role 
of  the  Government  Program  Manager  to  assure  that  sufficient  analysis  has  been 
performed,  prior  to  design  reviews,  which  demonstrates  with  a  high  degree  of  confidence 
that  diagnostic  requirements  are  being  met. 

A.  Scheduling 

It's  either  too  early  or  too  late.  Picking  the  optimum  time  for  reviews  is  very 
important.  Reviews  need  to  be  conducted  after  the  design  is  sufficiently  defined  to  make 
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the  evaluation  (the  type  of  evaluation  being  deternilned  by  the  type  of  roviovy),  b'/t  befo*"®  it 
is  too  late  to  make  changes. 

The  only  identified  lesson  learned  from  experience  is  that  the  scheduling  for 
formal  reviews  is  typically  performed  at  the  beginning  of  the  program.  The  stage  of  the 
design  for  the  review  is  then  whatever  it  is  at  the  scheduled  time.  This  is  not  necessarily 
bad,  because  typically  the  designers  work  toward  having  a  reviewable  product  on  the 
established  schedule.  Usually,  reviews  cannot  be  delayed  without  jeopardizing  delivery 
schedules. 

B.  Format 

Diagnostics  usually  involve  the  "whole"  picture.  This  should  be  considered  in 
developing  the  review  agenda.  Prior  to  the  diagnostics  portion  of  the  review,  a  full 
understanding  of  the  hardware/software  design  and  maintenance  concept  is  required. 

C.  Government  Emphasis 

Messages  are  sent  to  the  contractor  telling  him  where  he  should  put  his 
emphasis,  based  on  the  importance  an  item  has  in  the  review.  If  the  Government 
Program  Manager  and  Ns  review  team  place  little  emphasis  on  diagnostics,  the  contractor 
gets  that  message  and  typically  follows  suit.  A  sure  fire  way  to  hinder  the  progress  of  the 
diagnostic  design  team  is  to  indicate  to  the  Contractor  Program  Manager  that  diagnostics 
are  "not  important."  This  has  often  been  done  uninientionally,  in  the  past,  by  quickly 
passing  over  the  subject  in  the  review,  or  otherwise  indicating  a  minimal  concern. 

VI.  Demonstrations 

Demonstrations  are,  in  general,  another  form  of  a  formal  review.  Thus  most  of 
the  points  made  in  the  previous  section  also  apply  here. 

A.  Timeliness 

The  opportune  time  for  final  demonstration  of  diagnostics  does  not  exist,  if  a 
purpose  of  the  demonstration  is  to  identify  corrective  actions.  Efforts  to  schedule 
demonstrations  early  enough  to  minimize  the  Impact  of  "failure"  have,  in  the  past,  resulted 
in  the  simulation  of  too  many  conditions  and  resources.  To  perform  a  complete 
diagnostics  demonstration,  all  operational  diagnostics  tools  must  be  in  place.  This 
includes  support  equipment--if  appropriate-training,  technical  publications,  and  any  other 
applicable  diagnostic  tool.  Attempts  to  simulate  or  work  around  the  absence  of  these 
operational  items  does  not  provide  for  a  complete  demonstration. 
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B.  Simulated  vs.  Operational  Conditions 

The  problem  can  be  demonstrated  by  experience  with  a  recent  modification 
program  on  the  F-1 1 1D  Attack  Radar.  The  modification  was  major-mainly  made  to 
improve  reliability  and  maintainability.  One  significant  portion  of  the  modification  was  the 
rework  of  the  built-in  test  (BIT)  capabilities. 

*  « 

The  design  job  seemed  to  be  done  very  well.  Design  Reviews  were  passed. 
Demonstrations  of  the  new  BIT  performance  in  the  laboratory  exceeded  the  specifications 
and  expectations.  All  looked  like  a  job  well  done,  and  the  contract  was  considered 
complete. 


The  problem  was  that  on  the  aircraft,  in  operational  conditions,  the  BIT  does  not 
do  so  well.  The  BIT  serves  two  functions,  one  being  to  advise  the  aircrew  if  the  selected 
mode  is  operational,  the  other  serving  as  a  diagnostics  aid  to  maintenance  personnel. 
The  aircrew  function  performs  well,  which  is  not  surprising,  being  part  of  the  basic 
operational  requirement.  However,  the  diagnostics  portion  of  the  software  used  in  the  fauit 
isolation  process  has  required  extensive  rework.  At  first  glance,  one  is  led  to  believe  that 
the  simulated  and  operational  conditions  must  differ  greatly.  This  being  the  case,  how 
does  one  explain  that  problems  reported  during  field  operations  can  later  be  demonstrated 
under  laboratory  conditions?  Simulated  conditions  may  in  fact  create  problems  at  times, 
but  like  in  this  case,  why  do  they  generally  appear  in  the  area  of  diagnostics?  Performing 
demonstrations  with  the  primary  objective  of  showing  operational  requirements  are  being 
fulfilled,  with  diagnostics  given  secondary  concern  only  delays  finding  problems  in  that 
area.  An  important  point  to  remember  is  that  diagnostics  must  be  given  equal 
consideration  to  operational  requirements  and  the  Demonstration  Phase  is  another 
chance  to  identify  this  problem. 

At  this  point,  money  has  not  been  allocated  to  find  out  why.  The  point  is  that 
there  are  external  influences  which  are  often  not  understood.  The  only  way  they  can  be 
identified  is  to  test  the  end  item  under  the  conditions  where  it  is  to  be  used. 

If  time  and  money  do  not  provide  for  a  fully  operational  demonstration,  it 
becomes  the  hope  of  the  Government  Program  Manager  that  there  are  no  unknowns 
iurking  in  the  operationai  environment.  This  is  indeed  possible.  If  the  design  team  was 
knowledgeabie  enough  in  its  efforts,  success  will  be  at  hand. 

C.  Providing  for  Resources 

Scheduiing/obtaining  resources  for  the  demonstration  is  an  early  program 
function.  This  requirement  has  often  been  overlooked  or  minimized  in  the  past. 
Government  Program  Managers  need  to  be  fuliy  aware  of  the  demonstration 
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plan/requirements  and  assure  that  they  are  included  in  the  very  earliest  top-level  planning 
documents. 

VII.  MATURATION 

Maturation  is  a  phase  which  has  been  identified  as  necessary  primarily  during 
development  of  new  systems/technology  for  the  embedded  and  external  diagnostic 
capability.  One  especially^'critical  area  for'these  systems  is  the  inherent  requirement  for 
testing  under  actual  operating  conditions.  Maturation  becomes  necessary  to  refine  test 
method/fault  limits/diagnostics  logic  embedded  within  the  diagnostic  software  programs 
that  operate  these  systems.  The  predicted  operating  characteristics  of  the  various  on¬ 
board  systems  must  be  compared  to  the  operation  of  these  systems  as  they  interface  with 
other  systems  and  during  varying  environmental  conditions. 

A.  Early  Planning 

The  Government  Program  Manager  lesson  demonstrated  by  B-1B  experience  is 
to  plan  for  considerable  time  and  resources  to  allow  maturation.  The  original  B-1B 
development  plan  was  to  mature  the  diagnostic  system  (CITS)  on  70  PSD  flights.  That 
would,  it  was  thought,  provide  a  mature  system  at  the  time  of  the  first  deployment  of  the  B- 
1B  to  an  Air  Force  Main  Operating  Base.  Early  in  the  Full-Scale  Development  Phase,  it 
became  evident  that  that  plan  would  not  be  sufficient.  A  new  plan  was  developed  to  use 
468  SAC  sorties  over  the  years  1 985  and  I986.  The  wing  did  not  fly  the  required  number 
of  sorties  over  that  time  period  and  the  program  was  extended  through  November  1 987. 
Additional  aircraft  deliveries  and  an  increase  in  sortie  generation  rate  produced  a  total  of 
1060  sorties  by  the  end  of  that  period.  With  that  number  of  sorties,  sufficient  data  had 
been  gathered  to  indicate  a  more-than-acceptable  level  of  performance.  At  this  point,  it  is 
estimated  that  as  a  general  rule,  at  least  400  to  500  sorties  will  be  required  to  mature  an 
on-board  test  system  like  the  CITS. 

B.  Operational  or  Flight  Test  Environment 

How  does  one  plan  for  500  sorties  prior  to  production?  Is  a  plan  to  fly  four  FSD 
aircraft  on  the  average  of  once  every  three  calendar  days  for  a  year  reasonable?  Is  a 
limited  production  block  appropriate  for  maturation?  These  are  questions  for  which  the 
Government  Program  Manager  must  get  answers  early  in  program  planning. 

Experience  has  identified  one  additional  consideration  to  be  included  in  making 
these  planning  decisions.  That  consideration  is  the  impact  a  partially  working  diagnostic 
system  has  on  the  maintenance  technician.  If  technicians  lose  confidence  in  a  diagnostic 
aid,  they  will  not  use  it.  Further,  it  is  hard  to  convince  them  that  the  item  has  been 
improved  and  that  now  they  can  have  confidence  in  it.  Many  maintenance  technicians  on 
the  B-1 B,  F-1 1 1 ,  and  other  systems  who  have  been  exposed  to  inaccurate  diagnostic 
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methods  have  never  been  convinced  to  use  an  "improved"  version.  All  B-1 B  operating 
bases  have  the  same  current  version  of  CITs.  Field  data  shows,  however,  that  the  bases 
exposed  to  the  earliest  and  poorest  version  of  CITS  continue  to  have  the  highest  false 
alarm  and  cannot  duplicate  maintenance  rates.  This  is  due  to  the  lack  of  trust  still  carried 
from  the  early  experience.  Thus  it  is  important  to  arrange  for  maturation  away  from  the 
majority  of  operational  technicians,  if  possible. 

C.  Iinplementing  Maintenance  Concept 

Special  training  will  need  to  be  provided,  if  the  maintenance  concept  utilizing  the 
planned  diagnostics  is  significantly  different  from  that  with  which  the  established  technician 
is  familiar.  The  B-1  B  conflict  between  using  CITS  or  system  performance  to  rule  a  failure 
was  discussed  in  Section  III  of  this  Appendix.  Trends  are  also  in  place  today  to  isolate  to 
and  replace  modules  on  the  aircraft  rather  than  the  large  "boxes"  of  the  past.  Utilizing  the 
diagnostic  indication  produced  during  the  flight,  without  further  ground  verification,  is  also 
a  current  trend.  Each  of  these  "new"  concepts  must  be  thoroughly  understood  by  tf.e 
technicians,  so  that  the  maturation  results  are  consistent  with  the  planned  fielded 
maintenance  concept. 

VIII.  SUMMARY 

Diagnostics  must  be  a  simple  matter,  since  everything  always  works  at  the  end 
of  the  program.  However,  this  is  not  the  case,  and  the  perfect  situation  portrayed  in  the 
Introduction  has  yet  to  be  achieved.  Instead  of  the  capability  to  identify  the  one  LRU, 
often  the  ambiguity  group  is  as  many  as  four  LRUs.  The  ATE  which  can  isolate  ♦he  failure 
to  a  single  failed  component  would  be  the  ideal  solution,  but  more  likely  than  not,  it  will 
only  be  to  one  or  sometimes  several  shop  replaceable  units  (SRUs)  or  a  particular  group 
of  SRUs.  The  steps  covered  here  are  only  some  of  the  very  basic  ones  required  to  ensure 
good  diagnostics.  However,  looking  at  many  different  programs,  one  finds  even  these 
simple  steps  have  been  omitted,  or  perhaps  accomplished  at  a  time  too  late  to  have  the 
desired  results.  The  reasons  are  many:  poor  communication  of  needs  or  goals,  time 
frame  restrictions,  money,  and  failure  to  properly  consider  the  importance  of  diagnostics. 
To  ensure  diagnostics,  it  must  be  addressed  at  all  phases  and  be  given  equal  importance 
to  other  performance  requirements.  If  the  system  cannot  be  maintained,  it  can  never  meet 
its  operational  requirements. 
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O'  The  government  needs  one  specific  person  to  manage 
the  diagnostic  activities.  For  programs  such  as  the 
B— IB,  this  should  be  his  only  function. 

of  The  contractor  also  needs  one  person  at  the  program 
management  level  for  diagnostics.  This  should  be  his 
only  concern. 

The  persons  filling  the  above  positions  need  special 
skills  over  those  of  being  a  good  manager.  These 
include  an  understanding  of  logistics,  system  design, 
and  diagnostics. 

O'  The  persons  filling  the  above  positions  need  to 

communicate  with  each  other  and  with  their  respective 
organizations  on  a  frequent  basis. 

of  Studies,  analyses,  and  feedback  take  time.  They  need 
to  be  scheduled  so  that  their  results  can  influence 
the  design. 

Test  equipment  designers  need  to  have  an  input  regard¬ 
ing  the  design  requirements  of  the  units  to  be  tested. 

E2f  Proper  priorities  need  to  be  demonstrated  by  all  levels 
of  management  on  both  sides.  If  diagnostics  is  impor¬ 
tant,  all  management  must  consider  it  important. 

E2f  Specifications  must  be  well  defined  and  represent 
exactly  what  is  needed. 

of  Real  operating  time  is  required  for  maturation  of 
the  diagnostic  system — lots  of  it. 
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LESSONS  LEARNED 
RETRORTTING  AN  EXISTING  SYSTEM 


I.  INTRODUCTION 

This  guide  stresses  the  need  to  ”design-in"  the  weapon's  diagnostic 
capabiiity  as  the  design  of  the  prime  hardware  and  software  progresses.  The 
concept  being  "do  it  right  the  first  time."  Unfortunately,  the  possibility  of  addressing 
the  design  of  the  diagnostic  capability  beginning  at  Concept  Exploration  and 
following  through  to  Deployment  is  becoming  less  and  less  possible.  New  starts 
are  becoming  less  frequent  and  modifications  to  existing  systems  more  prevalent. 
In  addition,  retrofits  to  existing  weapon  system  diagnostic  capabilities  will  continue 
to  be  required.  The  capability  to  design  or  improve  the  diagnostic  capability, 
beginning  at  any  point  in  the  acquisition  or  deployment  of  a  weapon  system,  is  a 
must.  Thus,  the  following  example  of  the  M>1  Abrams  tank. 

Deficiencies  in  the  diagnostic  capability  of  the  M-1  led  to  the  initiation  of 
an  Integrated  Diagnostics  Improvement  Program  which  is  a  positive  example  for 
improving  the  diagnostic  capability  after-the-fact.  As  with  the  M-1 ,  dissatisfaction 
with  a  fielded  diagnostic  capability  is  not  limited  to  just  the  user.  It  becomes  a 
technology  issue,  a  cost  issue,  and  certainly  a  political  issue.  In  the  case  of  the  M- 
1 ,  the  maintenance  issue  became  a  national  issue,  draped  in  negatives. 


©1986  -  King  Features  Syndicate,  Inc.  Reprinted  with  special 
permission  of  King  Features  Syndicate,  Inc. 

To  develop  and  implement  the  necessary  improvements,  a  Joint  Working 
Group  for  Integrated  Diagnostic  Improvements  was  formed  with  a  charter  to 
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develop  a  system  engineering  approach  to  the  improvement  and  integration  of  its 
diagnostic  capabiiity.  For  an  existing  system  with  diagnostic  problems,  redesign  of 
the  system  or  subsystems  to  include  testability  is  a  potential  solution.  Other 
potential  solution  areas  include  improvement  of  the  fielded  diagnostic  elements, 
such  as  test  equipment,  test  programs,  test  procedures,  technical  information, 
maintenance  aids,  maintenance  personnel  skill  levels,  etc.  The  costs  and  benefits 
of  these  potential  solutions  must  be  carefully  analyzed.  Integration  among  the 
diagnostic  elements  also  must  be  maintained  or  achieved. 


II.  SUMMARY  OF  DIAGNOSTIC  PROBLEMS 

The  first  order  of  business  for  the  Joint  Working  Group  (JWG)  was  the 
development  of  integrated  diagnostic  improvement  objectives  and  development  of  a 
roadmap  to  achieve  the  objectives.  The  JWG  objectives  are  listed  below: 

o  Resolve  diagnostic  problems  related  to  support  of  Abrams 

o  Identify  cost-effective  solutions  to  the  problem{s)  that  complement 
short-term  fixes 

o  Ensure  that  the  diagnostic  concept  supports  future  Air  Land  Battle 
Doctrine 

o  Communicate  lessons  learned  to  combat/material  developers. 

The  JWG  defined  their  scope  of  effort  to  include  the  man,  test  equipment, 
technical  information,  training  and  tank  interfaces  (for  embedded  and 
nonembedded  diagnostics). 

The  JWG  developed  a  list  of  Abrams  problems  related  to  diagnostics. 
The  JWG  categorized  problems  according  to  embedded  and  nonembedded 
diagnostic  elements.  Prioritized  embedded  diagnostic  problems  are  presented 
below.  Limited  Built-In  Test  (BIT)  for  the  entire  vehicle  is  the  most  serious 
embedded  diagnostic  deficiency.  The  extent  of  the  BIT  deficiency  is  depicted  in 
Figure  8,  M-1  Tank  Built-In  Test.  The  mobility  system  does  not  have  an  embedded 
diagnostic  capability.  For  the  remainder  of  the  tank  electronic  systems,  on-line  fault 
detection  coverage  is  only  46  percent.  The  majority  of  BIT  coverage  fault  Isolates 
to  subsystem  functional  ambiguity  groups.  Embedded  off-line  fault  isolation 
coverage  includes  22  LRUs,  but  only  fault  isolates  to  the  single  LRU  27  percent  of 
the  time.  BIT  fault  isolation  to  the  single  LRU  Is  limited  to  the  Laser  Range  Finder 
and  the  Thermal  Imaging  Unit.  In  most  cases,  nonembedded  diagnostics  are  still 
required  to  fault  isolate  to  a  single  LRU. 
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Diagnostic  probiems  are  presented  beiow.  A  common  misconception  is 
that  the  Simpiified  Test  Equipment  (STE)  core  hardware  is  the  root  of  aii  diagnostic 
problems.  The  real  culprit  is  tank  diagnostic  design  which  contains  over  1 00  unique 
cable  connectors  resulting  in  the  proliferation  of  STE  Test  Program  Set  (TPS) 
cables  and  connectors. 

ABRAMS  DIAGNOSTIC  PROBLEM  SUMMARY 
EMBEDDED  (TANK) 

1 .  Limited  BIT  for  entire  vehicle 

a.  No  embedded  diagnostics  for  mobiiity  system 

b.  No  BIT  for  sensors 

2.  Not  an  integrated  diagnostic  system 

3.  No  standard  diagnostic  connector  assemblies 

4.  Cannot  fault  isolate  to  a  single  LRU  or  cable  (Non-intrusively) 

5.  No  data  recording  capability 

a.  Intermittent  faults 

b.  Historical 

6.  Poor  LRU,  cable  and  connector  accessibility 

7.  No  standard  test  connector  design 

8.  Limited  embedded  sensors  for  vehicle  transmission 

9.  Limited  space  for  new  hardware 


NONEMBEDDED  DIAGNOSTICS 
-  STE  M-l  (specific  to  the  M-1) 

Lack  of  Effectiveness  in  the  Operational  Environment 

STE  >  CORE  (Generic) 

1 .  Does  not  meet  environmental  requirements 

2.  Display  not  visible  in  direct  sunlight 

3.  Self-test  takes  too  long 

STE  -  TEST  PROGRAM  SETS 

1 .  Requires  cumbersome  cable  connections  (bulky) 

2.  Technical  manuals  not  written  for  use  as  TPS  documentation  with 
STE  (independent  action) 

3.  Test  Strategy 
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a.  Examines  failure  from  component  level  rather  than  system  level 

b.  Stops  after  first  fault  identified 

c.  Restart  to  recover  from  operator  error 

d.  Experienced  mechanic  cannot  eliminate  test  steps 

e.  Does  not  take  advantage  of  known  fault  data  (e.g.,  from  BIT) 

4.  Test  times  do  not  support  doctrinal  limits 

5.  Test  messages  subject  to  misinterpretation 

6.  Test  measurements  not  provided  to  mechanic 

TECHNICAL  MANUALS 

1 .  Not  user  friendly 

a.  Volume 

b.  Excessive  cross  referencing 

2.  Serial  symptomatic  approach  to  diagnostics  cumbersome  and  time 
consuming 

3.  Paper  technical  manuals  not  suited  for  field  use 

4.  Written  only  to  lowest  skill  level 

5.  Time  consuming  to  make  changes/difficult  to  control  process 

TRAINING/PERSONNEL 

1 .  Mechanics  lack  theoretical  knowledge  base  (back  to  basics) 

2.  Mechanics  lack  sufficient  advanced  and  sustainment  training 


III.  THE  SYSTEM  ENGINEERING  APPROACH  TO  ABRAMS 

INTEGRATED  DIAGNOSTICS  IMPROVEMENT 

In  addition  to  developing  a  list  of  Abrams  diagnostic  problems,  the  JWG 
identified  24  diagnostic  improvement  programs.  The  question  the  JWG  had  to 
answer  was:  If  the  problems  are  well  known  and  improvement  programs  are  under 
development,  why  is  there  still  a  diagnostic  problem  and  what  can  the  JWG  do  that 
has  not  been  done  before? 

The  answer  to  the  question  involves  approaching  diagnostic 
improvement  from  a  systems  engineering  and  integrated  diagnostics  perspective. 
The  JWG  discovered  that,  within  Army  Materiel  Command  (AMC),  existing 
diagnostic  improvement  programs  focused  on  different  problem  areas.  Some 
improvement  programs  covered  the  same  problem  areas,  other  improvement 
programs  did  not  adequately  address  any  problem  area.  The  JWG  determined  that 
if  they  were  to  add  anything  to  what  has  already  been  done,  a  systematic  approach 
to  the  problem  must  be  defined  and  implemented.  Their  approach  to  integrating 
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diagnostic  programs  focuses  on  treating  the  diagnostic  eiements  and  their 
interfaces  as  a  system  from  the  unit  through  the  Intermediate  levels  of 
maintenance. 

To  provide' a  comprehensive  and  cost-effective  plan  for  Improving  the 
Abrams  diagnostic  capability,  the  Group,  using  a  systems  engineering  approach, 
developed  the  integrated  diagnostic  methodology  shown  in  Figure  9.  Based  on 
operational  requirements,  the  Group  prioritized  the  tank’s  systems  and  subsystems 
at  the  different  levels  of  repair  and  prioritized  the  tank’s  systems  and  subsystems 
from  a  criticality  of  repair  point  of  view  at  the  different  levels  of  repair. 
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The  JWG  segregated  diagnostic  improvement  into  two  phases;  short¬ 
term  fixes  and  iong-term  soiutions.  This  was  necessary  to  adjust  recommendations 
to  the  Abrams  production  schedule. 

Short-term  fix  recommendations  are  limited  to  ongoing,  government  and 
industry,  funded  embedded  and  nonembedded  diagnostic  programs  achievable 
through  1990. 

The  JWG  recognized  that  short-term  fixes  could  not  possibly  address  all 
of  the  Abrams  diagnostic  problems.  The  Group  also  recognized  that  certain  design 
problems  could  not  be  cost-effectively  remedied  in  the  near  term,  i.e.,  no  standard 
test  connector  design,  limited  embedded  sensors  for  vehicle  transmission  and  poor 
LRU.  cable  and  connector  accessibility.  These  types  of  problems  will  be  addressed 
in  the  long-term  solutions  plan. 

The  Group  chose  a  user  perspective  to  narrow  the  set  of  Abrams 
diagnostic  problems  to  a  manageable  subset  for  short-term  solutions.  From  a  users 
perspective,  what  needs  improvement  the  most  (non-prioritized)  includes: 

o  Provide  embedded  diagnostic  capability  for  mobility  system:  In  a 
battle  zone,  an  immobile  tank  is  a  dead  tank.  The  ability  to  rejoin  the 
battle  or  limp  home  is  impaired  by  the  fault  detection  and  fault 
isolation  time  required,  and  by  the  vulnerability  and  limited  quantity  of 
maintenance  teams  available  to  support  a  tank  company.  Embedded 
diagnostics  in  the  tank  which  can  detect  and  isolate  a  mobility  fault 
without  outside  intervention  reduces  diagnostic  time  to  the  minimum 
and  provides  each  tank  with  its  own  "built-in  diagnostic  team." 

o  Reduce  size  and  weight  of  technical  manuals  and  Tesi, 
Measurement,  and  Diagnostic  Equipment  (TMDE):  An  M-113  or  M-88 
does  not  have  the  capability  to  carry  mechanics,  spare  parts,  cases  of 
test  equipment,  cases  of  cables  and  connectors,  and  12  feet  of 
technical  manuals.  A  tank  at  the  down  site  cannot  be  repaired  without 
mechanics  and  spare  parts.  The  only  candidates  left  for  size  and 
weight  reduction  are  the  technical  manuals  and  TMDE. 

o  Reduce  dependency  on  technical  manuals  and  STE:  In  the  Abrams 
diagnostic  history,  field  workarounds  were  adopted  to  circumvent  use 
of  the  STE  and  technical  manuals.  The  dedicated  contractor  support 
mechanics  did  not  depend  on  technical  manuals  and  the  STE.  They 
were  able  to  diagnose  faults  based  on  their  expertise  with  the  aid  of 
simple  test  equipment,  system  functional  flow  charts  and  wiring 
diagrams.  The  senior  NCOs  and  Warrant  Officers  observing  these 


A-21 


LESSONS  LEARNED 


APPENDIX  A 


improvements  were  convinced  there  are  other  approaches  to 
diagnostics.  This  observation  led  to  the  creation  of  the  Master 
Diagnostician  (Master  "D")  concept.  The  Master  "D"  troubleshooting 
approach  reduces  the  dependency  on  technical  manuals  and  the 
STE,  allowing  the  maintenance  teams  to  carry  more  spare  parts.  It 
also  reduces  the  problem  of  trying  to  reference  a  paper  manual  in 
inclement  weather  and  reduces  diagnostic  time  by  eliminating  the 
requirement  to  connect  bulky  test  cables  and  connectors  to  the  tank. 

o  Reduce  requirements  for  personnel  possessing  Master  "D”  skills 
(expert  system  instead):  The  Master  "D"  is  capable  of  troubleshooting 
the  tank  with  limited  technical  information  and  test  equipment.  The 
Master  "D"  relies  on  a  superior  knowledge  of  operational  theory  and 
fault  isolation  procedures.  The  problem  with  Master  "D"  is  the  cost  of 
training,  limited  availability  of  candidate  mechanics  and  low  retention 
of  personnel  with  Master  "D”  skills.  Given  the  current  state  of  expert 
system  technology,  the  logical  solution  to  the  Master  "D"  problem  is  to 
incorporate  his  knowledge  of  the  tank  and  troubleshooting  procedures 
into  an  expert  system.  A  "Master  D  in  a  box"  eliminates  the 
requirements  for  a  mechanic  with  Master  "D"  skills,  reduces  training 
requirements  and  enhances  the  average  mechanics  skill  level  to  that 
of  a  Master  "D". 

o  Improve  speed  of  fault  diagnosis;  In  a  remove  and  replace 
maintenance  scenario  practiced  at  the  downsite  and  collection  point 
for  electrical  and  electronic  components,  fault  diagnosis  is  the 
bottleneck  to  system  repair.  Improvements  in  fault  detection  and  fault 
isolation  offer  the  greatest  payback  in  reducing  the  unit-level 
maintenance  burden.  Enhanced  speed  and  ease  of  diagnostics  at 
the  down  site  and  battalion  collection  point  will  discourage  use  of 
swing  maintenance  practices.  Arbitrary  removal  and  replacement 
causes  unacceptably  high  No-Evidence-of-Failure  (NEOF)  rates.  It 
also  increases  the  workload  on  the  battalion  maintenance 
organization  and  depletes  the  Prescribed  l  oad  List/  Authorized 
Stockage  List  (PLUASL)  inventory. 


IV.  THE  ABRAMS  DIAGNOSTIC  SYSTEM 

Ordnance  Center  and  Armor  School  JWG  representatives  rank  the  battle 
criticality  of  tank  subsystems  as  follows; 
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1.  Mobility 

2.  Firepower 

3.  Surveillance 

4.  Survivability 

5.  Communications. 

The  JWG  endorsed/recommended  ehort-range  programs  are  presented 
in  Figure  10.  Diagnostic  improvement  efforts  concentrated  on  the  top  three 
subsystems.  Mobility  is  covered  under  Hull  Embedded  Capability.  Firepower  and 
Surveillance  are  covered  under  Turrent  Embedded  Capability.  All  three 
subsystems  are  covered  under  Hull  and  Turrent  Embedded  Capability  Expansion 
and  Nonembedded  Capability. 

From  a  diagnostic  point  of  view,  embedded  and  nonembedded  diagnostic 
capabilities  are  all  encompassing.  Embedded  diagnostics  includes  Built-In  Test 
(BIT),  Built-In  Test  Equipment  (BITE),  and  fault  data  recording  functions  to  support 
prognostics.  Recommended  short-range  embedded  programs  are  retrofitable  to 
existing  Abrams  tanks.  With  the  addition  of  an  optional  1553  multiplex  data  bus, 
short-range  embedded  programs  are  upwardly  compatible  with  Vetronics 
architecture  requirements. 

Nonembedded  diagnostic  recommendations  improve  current 
troubleshooting  procedures  and  lead  to  the  development  of  expert  system 
technology  to  improve  the  average  mechanics  troubleshooting  capability. 
Recommended  changes  to  the  nonembedded  test  strategy  decrease  diagnostic 
time  and  limit  the  use  of  cumbersome  test  equipment.  Nonembedded  diagnostics 
procedures  begin  with  the  simplest  non-intrusive  tests  possible  and  progress  to  the 
use  of  intrusive  tests  possible  and  progress  to  the  use  of  intrusive  test  equipment 
as  a  last  resort. 

The  nonembedded  troubleshooting  process  starts  with  the  results  of 
embedded  diagnostic  capabilities.  For  instance,  if  embedded  diagnostics  fault 
isolates  to  an  ambiguity  group  of  three  of  five  modules  in  a  system,  nonembedded 
diagnostic  should  start  the  troubleshooting  procedure  by  checking  out  the  three 
identified  modules.  This  does  not  always  occur  with  current  troubleshooting 
procedures.  Next,  the  mechanic’s  senses  (sight,  touch,  and  smell)  are  used  to 
locate  catastrophic  and/or  battle  damaged  components.  If  further  fault  isolation  is 
required,  a  Break-Out-Box  (BOB)  and  a  multimeter  are  employed  for  improved 
access  to  measure  signals  and  compare  to  good  known  values.  Finally,  if  all  other 
techniques  fail,  the  STE  is  used  to  determine  the  cause  of  failure. 
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The  following  sections  discuss  the  recommended  fixes  for  each 
diagnostic  area  presented  In  Figure  10,  after  considering  a  number  of  different 
alternatives. 


FIGURE  10.  Joint  Working  Group  Endorsed/Recommended  Programs 
(Short  Range) 
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A.  HULL  EMBEDDED  CAPABILITY 

1.  OBJECTIVE 

Since  mobility  is  the  key  to  a  tank’s  survival,  the  top  priority  of  hull 
diagnostics  is  to  provide  embedded  diagnostic/prognostic  capability  to  diagnose 
engine  problems.  Transmission  diagnostics  are  not  addressed  due  to  the  expense 
and  complexity  of  integrating  new  transmission  sensors  into  existing  tanks. 

2.  RECOMMENDED  FIX 

The  recommended  fix  for  engine  diagnostics  is  the  digital  Electronic 
Control  Unit  (ECU)  embedded  in  the  digital  Drivers  Instrument  Panel  (DIP),  which 
can  monitor  engine  system  signals  continuously  and  record  out-of-limit  signals 
continuously.  The  form,  fit,  and  function  permits  retrofit.  The  digital  ECU  promises 
to  provide  significant  reductions  in  O&S  cost. 

B.  TURRENT  EMBEDDED  CAPABILITY 

1.  OBJECTIVE 

Improvement  in  Stabilization  System  diagnostics  is  the  prime  objective  for 
torrent  embedded  diagnostic  capability. 

The  Laser  Range  Finder  and  Thermal  Imaging  Unit  contain  BIT  which 
fault  isolates  to  the  LRU.  The  complexity  of  the  Fire  Control  System  and 
Stabilization  System  prohibits  all  but  the  most  experienced  mechanics  (senior  NCO) 
and  Master  "D"s  from  troubleshooting  these  systems  without  the  use  of  the  STE. 
The  initial  STE  test  procedure  requires  the  STE  multiple  branch  adapter.  This 
adapter  is  cumbersome,  bulky,  and  difficult  to  connect  to  the  system.  Elimination  of 
the  multiple  branch  adapter  requirement  will  improve  diagnostic  time  by  reducing 
test  hook-up  and  run  time. 

2.  RECOMMENDED  RX 

The  recommended  fixes  consist  of  both  short-  and  long-term  initiatives. 
For  the  short  term,  recommended  continued  development  and  fielding  of  the  STE 
multiple  entry  point  test  program.  This,  together  with  other  STE  enhancements,  will 
reduce  test  times  and  increase  soldier  acceptance  of  STE  in  troubleshooting  the 
Stabilization  System.  The  iong-term  soiution  is  to  incorporate  Buiit-in  Test  (BIT)  for 
the  Stabilization  System.  For  any  fault  not  covered  by  BIT,  recommend  use  of  the 
Break-Out-Box  and  the  Master  Fault  procedures.  STE  testing  of  the  Stabilization 
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System  will  be  retained  only  if  it  is  determined  that  BIT  and  Master  Faults  do  not 
provide  100%  fault  coverage. 

C.  HULL  AND  TURRENT  EMBEDDED  CAPABILITY  EXPANSION 

1.  OBJECTIVE 

The  recommendations  presented  in  Sections  (A)  (Hull)  and  (B)  (Turrent) 

cover  the  most  essential  subsystems  and  most  pervasive  diagnostic  problems 
encountered  In  the  tank.  However,  the  above  recommendations  do  not  cover  all 
tank  LRUs.  The  objective  of  this  section  is  to  determine  the  embedded  diagnostic 
requirements  for  the  remaining  LRUs. 

2.  RECOMMENDED  FIX 

The  recommended  approach  to  accomplishing  this  objective  is  based  on 
a  systems  engineering  methodology.  The  methodology  provides  an  objective 
means  of  determining  which  of  the  remaining  LRUs  (if  any)  are  candidates  for 
redesign  to  increase  their  embedded  diagnostic  capability. 

The  Chrysler  Huntsville  Automotive  Tank  and  Engine  Prognostics  System 
(ATEPS)  Program  proposes  a  comprehensive  redesign  of  tank  LRUs  to  improve 
reliability  by  digitizing  LRU  functions,  to  increase  BIT  capability,  and  to  provide  a 
standard  bus  architecture.  A  complete  ATEPS  design  may  be  too  expensive  to 
implement.  However,  it  may  be  cost  effective  to  selectively  implement  specific 
ATEPS  LRUs. 

The  systems  engineering  approach  to  identifying  LRUs  for  redesign 
includes  the  following  steps; 

1 .  List  LRUs  which  cannot  be  fault  isolated  by  embedded  diagnostics 

2.  Rank  the  LRUs  by: 

-  MTBF 

-  MTTR 

-  NEOF  rate. 

3.  Select  ATEPS  LRUs  which  are  form-fit-function  replacement  for 
ranked  LRUs 

4.  Quantity  realistic  reliability  and  diagnostic  improvements  for  candidate 
ATEPS  LRUs 

5.  Perform  cost  analysis  for  each  candidate  using  the  TMDE  O&S 
Analysis  Model 

6.  Select  LRUs  for  redesign  based  on  potential  cost  avoidance. 
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D.  NONEMBEDDED  CAPABILITY 

1.  OBJECTIVE 

The  nonembedded  diagnostic  capability  improvement  objective  at  the 
down  site  and  collection  point  is  to  reduce  the  requirement  (dependency)  for  highly 
skilled  personnel.  The  approach  to  meeting  this  objective  also  addresses  the 
following  user  objectives: 

o  Reduce  requirements  for  personnel  possessing  Master  "D"  skills 

o  Reduce  size  and  weight  of  technical  manuals  and  test  equipment 

o  Reduce  dependency  on  technical  manuals  and  STE 

o  Improve  speed  of  fault  diagnosis 

o  Reduce  fault  isolation  ambiguity  groups  to  single  LRU  or  cable. 

2.  ALTERNATIVE  RXES 

The  following  alternatives  are  candidates  for  meeting  the  nonembedded 
diagnostic  objectives: 

o  MASTER  FAULTS  PROGRAM:  The  existing  technical  manual 
troubleshooting  approach  requires  extensive  cross  referencing  which 
is  both  time  consuming  and  confusing.  Manual-based  Master  Fault 
logic  will  minimize  these  conditions  through  two  features.  First,  the 
mechanic  is  required  to  identify  the  general  problem  area,  rather  than 
select  from  a  myriad  of  starting  points  corresponding  to  hundreds  of 
symptoms.  Second,  Master  Fault  logic  leads  the  mechanic  through  all 
visual  and  aural  performance  indications,  to  isolate  the  problem  to  a 
single  LRU  or  ambiguity  group  before  test  equipment  is  required  to 
complete  troubleshooting. 

o  MASTER  "D":  The  Master  "D"  Program  provides  advanced  diagnostic 
training  to  selected  senior  mechanics.  The  Master  "D"  is  highly  skilled 
in  unit-level  diagnostics,  troubleshooting  and  Battlefield  Damage 
Assessment  of  all  primary  weapon  systems. 

o  STE  TPS  Improvements: 

1.  STE  Test  Program  Set  Multiple  Entry:  The  STE  TPS  Multiple 
Entry  Is  a  demonstration  program  which  segments  the  existing  M-1 
Stabilization  Test  1400.  Multiple  entry  breaks  the  test  program 
into  segments.  Each  of  these  segments  can  be  executed 
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independently.  With  multiple  entry  points,  it  is  no  longer  necessary 
to  run  the  entire  program  from  beginning  to  end.  The  operator 
selects  the  segment  to  be  executed,  based  on  failure  symptoms. 
The  multiple  entry  Stabilization  Test  Program  is  intended  to  be  run 
without  the  multiple  branch  adapter. 

2.  STE  Test  Program  Set  Multiple  Faults:  Current  test  methodology 
requires  correction  of  each  fault  before  proceeding  with  a 
diagnostic  procedure.  The  fault  must  be  corrected  and  the  test 
reinitiated.  The  fault  may  represent  a  slight  "out-of-tolerance" 
measurement  not  significant  to  the  actual  fault  or  tank 
performance.  The  TPS  multiple  faults  methodology  will  store  all 
test  measurements  and  pass/fail  information  through  test 
completion.  The  STE  will  evaluate  all  test  results  and  provide  the 
operator  with  the  most  significant  faults  for  follow-on  corrective 
actions.  This  eliminates  current  requirement  to  restart  the  test 
program  after  each  fault  found. 

3.  STE  Test  Program  Recovery/Alert  Messages:  The  test  program 
alert  message  notifies  the  operator  when  a  measurement  has 
failed  "go  chain"  criteria  and  the  test  is  proceeding  down  a  "no-go" 
chain.  The  error  tolerant  recovery  allows  the  operator  to  retest  a 
failed  limit  measurement  when  notified  by  an  alert  message. 
When  the  retest  option  is  selected,  the  test  program  automatically 
branches  back  to  either  the  most  recent  block  of  operator  actions 
or  to  a  previousiy  unexecuted  diagnostic  word.  The  test  program 
then  provides  specific  recovery  instructions  before  resuming 
execution.  Alert  messages  and  error  recovery  allows  a  mechanic 
to  correct  operator  errors  without  reinitializing  the  test  program. 

4.  STE  Tost  Program  Set  -20  Technical  Manual  Automatic  Follow- 
Ons:  Present  STE  testing  methodology  results  in  fault  isolation  to 
a  single  LRU  or  to  an  ambiguity  group  of  LRUs  usually  including  a 
wiring  harness.  Whenever  the  initial  test  sequence  fault  isolates  to 
an  ambiguity  group,  a  fault  message  directs  the  operator  to  the 
technical  manual  for  instructions  and  follow-on  test  procedures 
which  identifies  the  next  test  program  to  execute  for  follow-on 
testing.  Incorporation  of  the  -20  automatic  follow-ons  will  allow  the 
operator  to  continue  with  STE  testing,  including  wiring  harness 
testing  until  a  single  fault  is  isolated  without  referring  to  the 
technical  manual.  This  reduces  technical  manual  references  and 
decreases  test  time. 
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o  STE  VIDEO  TRAINING  COURSE;  The  interactive  video  disc  program 
is  intended  to  teach  STE  and  -20  technical  manual  use  by  providing  a 
hands-on  training  situation  through  interactive  dialogue  with  the  video 
disc  program. 

o  STE  HEALTH  INDICATOR  TEST  DISPLAY  (HITD):  Sections  of  the 
engine  test  program  can  be  executed  to  check  the  health  of  the 
engine  in  a  preventive  maintenance  mode  where  no  faults  are 
present.  Future  maintenance  actions  can  be  prognosticated  based  on 
STE  HITD  results. 

o  -20  TECHNICAL  MANUAL  REFORMAT:  The  -20  technical  manuals 
are  being  reformatted  in  accordance  with  MIL-M-6038.  Due  to 
elimination  of  redundant  call-outs  and  pictures,  a  35-40%  volume 
reduction  is  anticipated. 

o  MECHANICS  HELPER:  The  scope  of  this  project  is  to  perform  a 
concept  study  involving  present  expert  system  technology  and  its 
application  in  the  field  of  diagnostics  and  maintenance,  incorporated 
in  this  study  is  an  expert  maintenance  overview  and  a  technology 
search.  This  study  will  be  used  to  examine  the  potential  of  expert 
maintenance  in  the  military  environment  and  to  determine  if  present 
expert  system  technology  is  capable  of  supporting  military 
maintenance  requirements.  Upon  successful  completion  of  this  study, 
the  program  will  proceed  into  a  limited  hardware  demonstration.  The 
hardware  demonstration  will  provide  a  working  knowledge  of  expert 
maintenance  systems  which  would  be  applied  toward  development  of 
a  Full-Scale  Engineering  Demonstration  System  capable  of  providing 
prognostic,  diagnostic,  and  repair  information  to  the  mechanic.  The 
expert  system  should  provide  the  mechanic  access  to  a  vast 
knowledge  base  which  could  locate,  in  a  few  seconds,  the  diagnostic, 
prognostic  and  repair  information  that  would  normally  take  several 
minutes  (or  hours)  to  locate  using  current  diagnostic  techniques.  The 
expert  system  can  be  hosted  on  the  Contact  Test  Set  (CTS)  being 
developed  under  the  Intermediate  Forward  Test  Equipment  (IFTE) 
Program. 


3.  RECOMMENDED  FIX 

Recommend  the  following  approach  to  reduce  the  requirement  for  highly 
skilled  personnel. 
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Even  though  the  Master  ”D”  is  the  epitome  of  highly  skilled  personnel,  he 
is  also  essential  to  the  short-term  diagnostic  solution.  His  diagnostic  techniques  do 
not  depend  on  bulky  technical  manuals  and  test  equipment  (uses  the  BOB, 
multimeter,  flow  charts,  and  wiring  diagrams).  The  Master  "D"  is  providing  and 
refining  the  Master  Fault  techniques.  This  is  an  important  step  towards  a  long-term 
solution  of  implementing  an  expert  system  based  on  Master  Fault  techniques. 

For  the  long-term,  the  Master  "D"  is  not  a  viable  program.  The  Master 
"D’s"  training  cost  are  prohibitive  (1 7  week  training  program).  Retention  rate  for  the 
Master  "D"  is  expected  to  be  low  due  to  promotion  to  Warrant  Officer  and  industry 
recruiting.  There  is  a  limited  pool  of  candidates  for  Master  "D"  training. 

The  STE  and  the  STE  TPS  improvements  are  endorsed  as  fixes  at  the 
Battalion  maintenance  echelon. 

The  long-term  solution  to  nonembedded  diagnostic  problems  is  the 
introduction  of  an  expert  system  such  as  the  Mechanics  Helper  to  the  diagnostic 
process.  The  expert  system  should  incorporate  the  Master  Fault  diagnostic 
procedures  tested  and  proven  by  the  Master  "D".  This  solution  eliminates  the 
reliance  on  technical  manuals,  lowers  initial  training  requirements,  and  provides  a 
built-in  tutor  for  advanced  and  sustainment  training.  Use  of  the  expert  system  by 
the  Advanced  Individual  Training  (AIT)  mechanic  guarantees  consistent 
maintenance  procedures  are  followed,  provides  consistent  diagnostic  results  and 
minimizes  intrusive  testing  at  the  down  site  and  collection  point.  Since  the  IFTE 
Contact  Test  Set  design  includes  an  expert  system  shell,  it  is  a  candidate  for 
demonstrating  the  Mechanics  Helper  capabilities. 

E.  SUMMARY 

In  summary,  the  systems  engineering  methodology  applied  to  the 
Abrams  diagnostics  improvement  resulted  in  multiple  solutions.  The  overall 
solution  involved/impacted  all  diagnostic  elements.  The  solution  focuses  on  the 
integration  of  the  diagnostic  elements  to  provide  a  comprehensive,  cohesive 
diagnostic  capability  within  the  constraints  of  an  existing  system  design  and  support 
design. 
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CHECKLIST 

Is  a  systems  engineering  approach  being  used  to 
modify/retrofit  the  diagnostic  capability? 


□f  Have  all  feasible  alternatives  been  considered 
and  evaluated? 


Have  emphasis  been  placed  on  correcting  the 
items  which  have  a  poor  diagnostic  history? 
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AFLC 

AFSC 

Al 

AIT 

AMC 

ATE 

ATEPS 

BIT 

BITE 

BOB 

C/ATLAS 

CAD 

CAE 

CALS 

CASS 

CDR 

CDRL 

CEPS 

CFE 

Cl 

CITS 

CND 

CNO 

CPCI 

CSC 

CSCI 

CSDM 

CSOM 

CTE 

CTS 

D-Level 

DBOO 

DCP 

DenrWal 

DFT 

DID 

DIP 


J 
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UST  OF  ACRONYMS 


Air  Force  Logistics  Command 

Air  Force  Systems  Command 

Artificial  Intelligence 

Advanced  Individual  Training 

Army  Materiel  Command 

Automatic  Test  Equipment 

Automotive  Tank  and  Engine  Prognostics  System 

Built-In  Test 

Built-In  Test  Equipment 

Break-Out-Box 

Common  Abbreviated  Test  Language  for  All  Systems 

Computer-Aided  Design 

Computer-Aided  Engineering 

Computer-Aided  Acquisition  &  Logistics  Support 

Consolidated  Automated  Support  System 

Critical  Design  Review 

Contract  Data  Requirements  List 

CITS  Expert  Parameter  System 

Contractor  Furnished  Equipment 

Configuration  Items 

Central  Integrated  Test  System 

Cannot  Duplicate 

Chief  of  Naval  Operation 

Computer  Program  Configuration  Item 

Computer  System  Component 

Computer  Software  Configuration  Item 

Computer  System  Diagnostic  Manual 

Computer  Software  Operator's  Manual 

Commercial  Test  Equipment 

Contact  Test  Set 

Depot  Level 

Data  Base  Design  Document 
Decision  Coordinating  Paper 
Demonstration  and  Validation  (Phase) 

Design  For  Testability 
Data  Item  Description 
Drivers  Instrument  Panel 
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DoD 

DoD-D 

DoD-INST 

DSESTS 

DT&E 

ECP 

ECU 

FA 

FCA 

FD 

FEFI 

FFI 

FFD 

FI 

FIS 

FMEA 

FMECA 

FOT&E 

FRACAS 

FSD 

FSM 

FYDP 

GFE 

GIMADS 

GPETE 

HW 

HWCI 

l-Level 

1C 

ID 

IDD 

IDSS 

IFTE 

ILS 

ILSP 

IMIS 

lOT&E 

IPS 

ITP 

JWG 


J 


Department  of  Defense 
DoD  Directive 
DoD  Instruction 

Direct  Support  Electrical  Systems  Test  Set 
Development  Test  and  Evaluation 

Engineering  Change  Proposal 
Electronic  Control  Unit 

False  Alarm 

Functional  Configuration  Audit 
Fault  Detection 

Fraction  of  Erroneous  Fault  Isolation  Results 

Fraction  of  Faults  Isolated 

Fraction  of  Faults  Detected 

Fault  Isolation 

Fault  Isolation  System 

Failure  Modes  and  Effects  Analysis 

Failure  Modes,  Effects  and  Criticality  Analysis 

Follow-On  Test  &  Evaluation 

Failure  Repeating  Analysis  and  Corrective  Action 

Full-Scale  Development 

Rrmware  Support  Manual 

Rve  Year  Defense  Plan 

Government  Furnished  Equipment 
Generic  Integrated  Maintenance  Diagnostics 
General  Purpose  Electi'onic  Test  Equipment 

Hardware 

Hardware  Configuration  Item 

Intermediate  Level 
Integrated  Circuit 
Integrated  Diagnostics 
Interface  Design  Document 
Integrated  Diagrrostics  Support  System 
Intermediate  Forward  Test  Equipment 
Integrated  Logistic  Support 
Integrated  Logistic  Support  Plan 
Integrated  Maintenance  Information  System 
Initial  Operational  Test  &  Evaluation 
Integrated  Program  Summary 
Integrated  Test  Plan 

Joint  Worthing  Group 
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LCC 

Life  Cycle  Cost 

LRM 

Line  Replaceable  Module 

LRU 

Line  Replaceable  Unit 

LSA 

Logistic  Support  Analysis 

LSAP 

Logistic  Support  Analysis  Plan 

LSI 

Large  Scale  Integration 

MASTER  "D" 

Master  Diagnostician 

MATE 

Modular  Automatic  Test  Equipment 

MD 

Maintainability  Demonst.ation 

MIL-STD 

Military  Standard 

MNS 

Mission  Need  Statement 

MTBF 

Mean  Time  Between  Failures 

MTE 

Manual  Test  Equipment 

MTTR 

Mean  Time  To  Repair 

NCO 

Non-Commissioned  Officer 

NDI 

Non-Developmental  Items 

NEOF 

No-Evidence-of-Failure 

NSIA 

National  Security  Industrial  Association 

0-Level 

Organizational  Level 

OJT 

On-the-Job  Training 

OT&E 

Operational  Test  &  Evaluation 

OUSD(A) 

Office  of  the  Under  Secretary  of  Defense  (Acquisition) 

p3| 

Preplanned  Product  Improvement 

PAT&E 

Production  Acceptance  Test  &  Evaluation 

PCA 

Physical  Configuration  Audit 

PDR 

Preliminary  Design  Re>^ew 

PLL/ASL 

Prescribed  Load  List/Authoiized  Stockage  List 

PMRT 

Program  Management  Responsibility  Transfer 

PPBS 

Planning,  Programming  and  Budgeting  System 

PROD/DEP 

Production/Deployment  (Phase) 

PRR 

Production  Readiness  Review 

RADC 

Rome  Air  Development  Center 

RDGT 

Reliability  Development/Growth  Test 

RFP 

Request  for  Proposai 

RISE 

Readiness  Improvement  Through  System  Engineering 

ROC 

Required  Operational  Capability 

RTOK 

Retest  OK 

SCP 

System  Coordinating  Paper 

SDDD 

Software  Detail  Design  Document 

SDR 

System  Design  Review 

SEMP 

System  Engineering  Management  Plan 
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SHARP 

SIT 

SON 

SOW 

SPM 

SRA 

SRR 

SRU 

STD 

STE 

STE  HITD 

STLDD 

STP 

SUM 

SW 

T&E 

T 

TAH 

TBD 

TEMP 

TFOM 

Tl 

TISSS 

TM 

TMDE 

TO 

TPI 

TPS 

TRD 

TRR 

UUT 

VHSIC 

VLSI 

WBS 

WRA 


Standard  Hardware  Acquisition  Requirement  Process 

System  Integrated  Test 

Statement  of  Need 

Statement  of  Work 

Software  Programmer’s  Manual 

Shop  Replaceable  Assembly 

System  Requirements  Review 

Shop  Replaceable  Unit 

Software  Test  Descriptions 

Simplified  Test  Equipment 

Simplified  Test  Equipment  Health  Indicator  Test  Display 

Software  Top-Level  Design  Document 

Software  Test  Plans 

Software  User’s  Manual 

Software 

Test  &  Evaluation 
Testability 

Testability  Analysis  Handbook 
To  Be  Determined 
Test  &  Evaluation  Master  Plan 
Testability  Rgure  of  Merit 
Technical  Information 

Tester  Independent  Support  Software  System 
Test  and  Maintenance 

Test,  Measurement,  and  Diagnostic  Equipment 

Technical  Orders 

Test  Program  Instruction 

Test  Program  Set 

Test  Requirements  Document 

Test  Readiness  Review 

Unit  Under  Test 

Very  High  Speed  Integrated  Circuit 
Very  Large  Scale  Integration 

Work  Breakdown  Structure 
Weapon  Replaceable  Assembly 
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MISSION 

of 

Rome  Air  Development  Center 


RADC  plans  and  conducts  research,  exploratory  and  advanced 
development  programs  in  conmand,  control,  and  communications 
(C^)  activities,  and  in  the  areas  of  information  sciences 
and  intelligence.  The  principal  technical  mission  areas 
are  communications,  electromagnetic  guidance  and  control, 
surveillance  of  ground  and  aerospace  objects,  intelligence 
data  collection  and  handling,  information  system  technology, 
ionospheric  propagation,  solid  state  sciences,  microwave 
physics  and  electronic  reliability ,  maintainabil ity  and 
compatibility . 
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